2024-04-25T21:26:34.150 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T21:26:34.150 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} --archive /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740 --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/orig.config.yaml 2024-04-25T21:26:34.186 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T21:26:34.269 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740 branch: squid description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} email: ceph-qa@ceph.com first_in_suite: false job_id: '7673740' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ mon_bind_msgr2: false sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9886 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 30270/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpKTVp25AOOPX/x1Pz8O7K0xHFoi+nouZxvb7b3qMPpNR7a0C6/1gczgfuh9x2uueTMajRol4aM3+Z25kf9EIw= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbLYmhHZNxHxKgC5WNhsun6mgkS/KmRQ1LXeyu+MFWgIWD0+lCQYDt6lgDIaouUANlA6ugDPBixcdaxOdWf99s= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - exec: client.0: - ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T21:26:34.269 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-25T21:26:34.270 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-25T21:26:34.270 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T21:26:34.272 INFO:teuthology.task.internal:Checking packages... 2024-04-25T21:26:34.295 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T21:26:34.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T21:26:34.295 INFO:teuthology.packaging:ref: None 2024-04-25T21:26:34.295 INFO:teuthology.packaging:tag: None 2024-04-25T21:26:34.295 INFO:teuthology.packaging:branch: squid 2024-04-25T21:26:34.295 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T21:26:34.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-25T21:26:34.520 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447-gb22e2ebd-1jammy 2024-04-25T21:26:34.522 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T21:26:34.529 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T21:26:34.529 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T21:26:34.542 INFO:teuthology.task.internal:Saving configuration 2024-04-25T21:26:34.554 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T21:26:34.561 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T21:26:34.586 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 21:20:45.745279', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE/ogHnMBLTZSjqxDC9tv2/eQJO0JaLY9rOi225P5vZF'} 2024-04-25T21:26:34.606 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 21:20:45.746531', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGQ6HDWkQY5YC9Yu50ismyTwWthSdikxe/hcJb1ORlP'} 2024-04-25T21:26:34.606 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T21:26:34.613 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-25T21:26:34.613 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-04-25T21:26:34.613 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T21:26:34.680 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1164c6e280>, signals=[15]) 2024-04-25T21:26:34.681 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T21:26:34.687 INFO:teuthology.task.internal:Opening connections... 2024-04-25T21:26:34.687 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:26:34.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:26:34.771 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:26:34.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:26:34.851 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T21:26:34.859 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-04-25T21:26:34.864 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-04-25T21:26:34.864 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-04-25T21:26:34.912 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="22.04" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=jammy 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T21:26:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=jammy 2024-04-25T21:26:34.914 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-04-25T21:26:34.939 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-25T21:26:34.944 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-25T21:26:34.944 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-25T21:26:34.990 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T21:26:34.990 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Ubuntu" 2024-04-25T21:26:34.990 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="22.04" 2024-04-25T21:26:34.990 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T21:26:34.990 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_CODENAME=jammy 2024-04-25T21:26:34.991 INFO:teuthology.orchestra.run.smithi203.stdout:ID=ubuntu 2024-04-25T21:26:34.991 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE=debian 2024-04-25T21:26:34.991 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T21:26:34.991 INFO:teuthology.orchestra.run.smithi203.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T21:26:34.991 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T21:26:34.991 INFO:teuthology.orchestra.run.smithi203.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T21:26:34.991 INFO:teuthology.orchestra.run.smithi203.stdout:UBUNTU_CODENAME=jammy 2024-04-25T21:26:34.991 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-25T21:26:35.014 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T21:26:35.023 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T21:26:35.028 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T21:26:35.029 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-04-25T21:26:35.031 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-25T21:26:35.037 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T21:26:35.043 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T21:26:35.043 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-04-25T21:26:35.078 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-25T21:26:35.093 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T21:26:35.147 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T21:26:35.160 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-25T21:26:35.160 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-25T21:26:35.161 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T21:26:35.161 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T21:26:35.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:26:35.165 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-25T21:26:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:26:35.167 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-25T21:26:35.213 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-56-generic 2024-04-25T21:26:35.213 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.15.0-56-generic 2024-04-25T21:26:35.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-04-25T21:26:35.215 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-56-generic 2024-04-25T21:26:35.215 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.15.0-56-generic 2024-04-25T21:26:35.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-04-25T21:26:35.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-04-25T21:26:35.347 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-04-25T21:26:35.488 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T21:26:35.488 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T21:26:35.492 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T21:26:35.511 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T21:26:35.512 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T21:26:35.512 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T21:26:35.585 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T21:26:35.599 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T21:26:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T21:26:35.750 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T21:26:35.796 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T21:26:35.828 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T21:26:35.834 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T21:26:35.841 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T21:26:35.849 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T21:26:35.851 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T21:26:35.852 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T21:26:35.882 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T21:26:35.883 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T21:26:35.889 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T21:26:35.889 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T21:26:35.894 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T21:26:35.900 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T21:26:35.901 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T21:26:35.901 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T21:26:35.902 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T21:26:35.912 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T21:26:35.914 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T21:26:35.922 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T21:26:35.942 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T21:26:35.947 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T21:26:35.947 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T21:26:35.950 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T21:26:35.950 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-25T21:26:35.956 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T21:26:35.959 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T21:26:35.959 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T21:26:35.961 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T21:26:35.961 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T21:26:35.962 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T21:26:35.968 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T21:26:35.970 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T21:26:35.986 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T21:26:35.987 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T21:26:35.991 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T21:26:35.991 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T21:26:35.993 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T21:26:35.993 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-25T21:26:35.994 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T21:26:35.994 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T21:26:35.994 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T21:26:35.998 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T21:26:36.002 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T21:26:36.003 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T21:26:36.004 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T21:26:36.005 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T21:26:36.014 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T21:26:36.017 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T21:26:36.017 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T21:26:36.018 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T21:26:36.030 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T21:26:36.030 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T21:26:36.031 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-25T21:26:36.031 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-25T21:26:36.032 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T21:26:36.034 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T21:26:36.040 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T21:26:36.040 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T21:26:36.050 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T21:26:36.052 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T21:26:36.052 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-25T21:26:36.053 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-25T21:26:36.053 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T21:26:36.053 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T21:26:36.062 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T21:26:36.064 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T21:26:36.065 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T21:26:36.065 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T21:26:36.066 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T21:26:36.067 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T21:26:36.067 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-25T21:26:36.068 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-25T21:26:36.086 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T21:26:36.087 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T21:26:36.108 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-25T21:26:36.108 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-25T21:26:36.109 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T21:26:36.109 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T21:26:41.516 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 13.2 MB in 2s (7,175 kB/s) 2024-04-25T21:26:41.926 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 13.2 MB in 2s (6,730 kB/s) 2024-04-25T21:26:42.995 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-04-25T21:26:43.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T21:26:43.072 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-04-25T21:26:43.271 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-04-25T21:26:43.272 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-04-25T21:26:43.337 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-25T21:26:43.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T21:26:43.416 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-25T21:26:43.437 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T21:26:43.437 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T21:26:43.437 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T21:26:43.438 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T21:26:43.438 INFO:teuthology.orchestra.run.smithi203.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T21:26:43.438 INFO:teuthology.orchestra.run.smithi203.stdout: python2.7-minimal 2024-04-25T21:26:43.438 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T21:26:43.439 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-04-25T21:26:43.439 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T21:26:43.439 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T21:26:43.440 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T21:26:43.441 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-04-25T21:26:43.441 INFO:teuthology.orchestra.run.smithi203.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T21:26:43.441 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-04-25T21:26:43.441 INFO:teuthology.orchestra.run.smithi203.stdout: thermald 2024-04-25T21:26:43.486 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-04-25T21:26:43.486 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T21:26:43.486 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T21:26:43.487 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T21:26:43.488 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages will be upgraded: 2024-04-25T21:26:43.488 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T21:26:43.604 INFO:teuthology.orchestra.run.smithi203.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T21:26:43.604 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 113 MB of archives. 2024-04-25T21:26:43.604 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T21:26:43.604 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T21:26:43.619 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-04-25T21:26:43.620 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-04-25T21:26:43.801 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T21:26:43.801 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T21:26:43.801 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T21:26:43.801 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T21:26:43.802 INFO:teuthology.orchestra.run.smithi006.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T21:26:43.802 INFO:teuthology.orchestra.run.smithi006.stdout: python2.7-minimal 2024-04-25T21:26:43.802 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T21:26:43.803 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-04-25T21:26:43.803 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T21:26:43.803 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T21:26:43.803 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T21:26:43.804 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-04-25T21:26:43.805 INFO:teuthology.orchestra.run.smithi006.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T21:26:43.805 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-04-25T21:26:43.805 INFO:teuthology.orchestra.run.smithi006.stdout: thermald 2024-04-25T21:26:43.858 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-04-25T21:26:43.858 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T21:26:43.858 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T21:26:43.858 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T21:26:43.860 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages will be upgraded: 2024-04-25T21:26:43.860 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T21:26:44.042 INFO:teuthology.orchestra.run.smithi006.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T21:26:44.043 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 113 MB of archives. 2024-04-25T21:26:44.043 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T21:26:44.043 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T21:26:44.436 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T21:26:44.695 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T21:26:45.428 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T21:26:45.792 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T21:26:45.987 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T21:26:45.987 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T21:26:45.987 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T21:26:46.692 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T21:26:46.886 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T21:26:47.288 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 113 MB in 3s (33.4 MB/s) 2024-04-25T21:26:47.441 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T21:26:48.036 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T21:26:48.036 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T21:26:48.037 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T21:26:48.389 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T21:26:48.472 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T21:26:48.873 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 113 MB in 5s (24.6 MB/s) 2024-04-25T21:26:49.058 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T21:26:49.477 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T21:26:49.481 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:26:49.546 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:26:51.085 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T21:26:51.089 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:26:51.181 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:26:51.538 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T21:26:51.551 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:26:51.647 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:26:52.116 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T21:26:52.137 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:26:52.175 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:26:53.641 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T21:26:53.663 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:26:53.762 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:26:54.202 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T21:26:54.223 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:26:54.260 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:26:58.612 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T21:26:58.740 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T21:26:59.082 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T21:26:59.219 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T21:26:59.497 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T21:26:59.520 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T21:26:59.564 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T21:27:00.645 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T21:27:00.746 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T21:27:01.089 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T21:27:01.233 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T21:27:01.528 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T21:27:01.551 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T21:27:01.594 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T21:27:05.072 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T21:27:05.108 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:27:05.146 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:07.442 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T21:27:07.480 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T21:27:07.517 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:08.058 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T21:27:08.180 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T21:27:08.506 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T21:27:08.627 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:08.753 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T21:27:08.882 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:10.199 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T21:27:10.199 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T21:27:10.199 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T21:27:10.199 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:10.245 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T21:27:10.352 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:10.370 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T21:27:10.738 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T21:27:10.858 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:11.407 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T21:27:11.905 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:12.096 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:13.237 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T21:27:13.237 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T21:27:13.237 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T21:27:13.237 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:13.391 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:13.480 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T21:27:13.598 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T21:27:13.724 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:13.789 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T21:27:13.789 INFO:teuthology.orchestra.run.smithi203.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:15.310 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:16.710 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T21:27:17.070 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T21:27:17.730 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T21:27:17.805 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T21:27:17.806 INFO:teuthology.orchestra.run.smithi006.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:27.837 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T21:27:27.837 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub' 2024-04-25T21:27:27.849 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T21:27:27.874 INFO:teuthology.orchestra.run.smithi203.stdout:Generating grub configuration file ... 2024-04-25T21:27:28.114 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T21:27:28.132 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:28.435 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T21:27:28.441 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T21:27:28.509 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T21:27:28.515 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T21:27:28.663 INFO:teuthology.orchestra.run.smithi203.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T21:27:28.663 INFO:teuthology.orchestra.run.smithi203.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T21:27:28.663 INFO:teuthology.orchestra.run.smithi203.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T21:27:28.703 INFO:teuthology.orchestra.run.smithi203.stdout:done 2024-04-25T21:27:29.183 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.105.102 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T21:27:29.201 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T21:27:29.202 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-04-25T21:27:29.202 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-04-25T21:27:29.202 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-04-25T21:27:29.202 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-04-25T21:27:29.202 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T21:27:29.202 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T21:27:29.202 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T21:27:29.202 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-25T21:27:29.202 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T21:27:29.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-04-25T21:27:29.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-04-25T21:27:29.478 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T21:27:29.627 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T21:27:29.701 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T21:27:29.784 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T21:27:30.867 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-04-25T21:27:30.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T21:27:30.942 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-04-25T21:27:31.143 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-04-25T21:27:31.144 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T21:27:31.315 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T21:27:31.355 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T21:27:31.357 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-04-25T21:27:31.371 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.105.102 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-04-25T21:27:31.372 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-04-25T21:27:31.373 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-04-25T21:27:31.417 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.wH7f78JycK 2024-04-25T21:27:31.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub/grub.cfg /tmp/tmp.wH7f78JycK 2024-04-25T21:27:31.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.wH7f78JycK 2024-04-25T21:27:31.589 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.wH7f78JycK is 10KB 2024-04-25T21:27:31.602 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.wH7f78JycK 2024-04-25T21:27:31.608 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T21:27:31.661 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:27:31.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T21:27:31.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T21:27:31.728 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T21:27:31.728 DEBUG:teuthology.orchestra.run.smithi203:> sudo update-grub 2024-04-25T21:27:32.220 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T21:27:32.220 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub' 2024-04-25T21:27:32.224 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T21:27:32.246 INFO:teuthology.orchestra.run.smithi006.stdout:Generating grub configuration file ... 2024-04-25T21:27:32.649 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T21:27:32.685 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:32.846 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub' 2024-04-25T21:27:32.847 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T21:27:32.849 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-04-25T21:27:33.027 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T21:27:33.034 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:33.073 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T21:27:33.080 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T21:27:33.148 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T21:27:33.154 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T21:27:33.275 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T21:27:33.279 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T21:27:33.317 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T21:27:33.321 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T21:27:33.341 INFO:teuthology.orchestra.run.smithi006.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T21:27:33.341 INFO:teuthology.orchestra.run.smithi006.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T21:27:33.341 INFO:teuthology.orchestra.run.smithi006.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T21:27:33.377 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-04-25T21:27:33.428 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T21:27:33.428 INFO:teuthology.orchestra.run.smithi203.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T21:27:33.428 INFO:teuthology.orchestra.run.smithi203.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T21:27:33.442 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-04-25T21:27:33.450 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-04-25T21:27:33.850 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-04-25T21:27:33.872 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-04-25T21:27:33.872 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-04-25T21:27:33.872 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-04-25T21:27:33.872 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-04-25T21:27:33.872 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-04-25T21:27:33.872 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T21:27:33.872 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.105.102 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-04-25T21:27:33.873 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-04-25T21:27:33.874 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T21:27:33.874 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T21:27:33.874 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T21:27:33.874 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-25T21:27:33.874 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T21:27:33.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-04-25T21:27:34.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-04-25T21:27:34.185 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T21:27:34.825 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T21:27:34.905 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T21:27:34.987 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T21:27:36.179 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-25T21:27:36.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T21:27:36.261 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-25T21:27:36.464 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-04-25T21:27:36.465 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-04-25T21:27:36.650 INFO:teuthology.orchestra.run.smithi006.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T21:27:36.650 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T21:27:36.650 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T21:27:36.650 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T21:27:36.650 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T21:27:36.650 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T21:27:36.651 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T21:27:36.651 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T21:27:36.694 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T21:27:36.695 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.105.102 2024-04-25T21:27:36.709 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T21:27:36.710 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T21:27:36.710 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-04-25T21:27:36.710 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-04-25T21:27:36.710 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-04-25T21:27:36.710 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-04-25T21:27:36.710 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-25T21:27:36.756 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.wU9jBlCXWd 2024-04-25T21:27:36.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub/grub.cfg /tmp/tmp.wU9jBlCXWd 2024-04-25T21:27:36.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.wU9jBlCXWd 2024-04-25T21:27:36.922 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.wU9jBlCXWd is 10KB 2024-04-25T21:27:36.934 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.wU9jBlCXWd 2024-04-25T21:27:36.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T21:27:36.992 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:27:36.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T21:27:36.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T21:27:37.060 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T21:27:37.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo update-grub 2024-04-25T21:27:38.358 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub' 2024-04-25T21:27:38.360 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T21:27:38.362 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-04-25T21:27:38.637 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T21:27:38.643 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T21:27:38.932 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T21:27:38.937 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T21:27:38.980 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T21:27:38.985 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T21:27:39.129 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T21:27:39.129 INFO:teuthology.orchestra.run.smithi006.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T21:27:39.129 INFO:teuthology.orchestra.run.smithi006.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T21:27:39.140 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-04-25T21:27:39.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-04-25T21:28:03.474 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-25T21:28:03.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T21:28:03.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:28:09.150 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-25T21:28:09.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-25T21:28:09.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:28:37.919 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-04-25T21:28:46.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T21:28:46.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:29:09.174 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T21:29:18.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-25T21:29:18.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:29:25.874 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-25T21:29:27.006 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-25T21:29:27.006 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-25T21:29:27.006 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-04-25T21:29:27.054 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-105-generic 2024-04-25T21:29:27.054 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T21:29:27.055 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T21:29:27.055 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T21:29:28.055 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T21:29:28.056 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T21:29:28.643 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-04-25T21:29:28.668 DEBUG:teuthology.parallel:result is None 2024-04-25T21:29:46.924 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T21:29:58.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T21:29:58.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:29:59.236 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-25T21:29:59.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T21:29:59.809 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-25T21:29:59.809 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-25T21:29:59.857 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-105-generic 2024-04-25T21:29:59.857 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T21:29:59.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T21:29:59.858 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T21:30:00.858 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T21:30:00.858 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T21:30:01.211 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-04-25T21:30:01.240 DEBUG:teuthology.parallel:result is None 2024-04-25T21:30:01.240 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T21:30:01.252 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T21:30:01.252 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T21:30:01.254 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T21:30:01.261 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T21:30:01.287 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T21:30:01.295 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T21:30:01.295 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T21:30:01.302 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T21:30:01.322 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T21:30:01.329 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T21:30:01.329 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T21:30:01.358 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T21:30:01.383 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T21:30:01.389 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T21:30:01.392 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T21:30:01.397 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T21:30:01.399 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T21:30:01.406 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T21:30:01.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T21:30:01.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T21:30:01.459 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T21:30:01.468 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T21:30:01.468 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T21:30:01.498 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T21:30:01.506 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T21:30:01.550 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T21:30:01.597 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:30:01.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T21:30:01.663 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T21:30:01.669 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T21:30:01.718 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:30:01.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T21:30:01.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-04-25T21:30:01.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-25T21:30:01.859 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T21:30:01.867 INFO:teuthology.task.internal:Starting timer... 2024-04-25T21:30:01.867 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T21:30:01.877 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T21:30:01.884 DEBUG:teuthology.task.selinux:Excluding smithi006: OS 'ubuntu' does not support SELinux 2024-04-25T21:30:01.885 DEBUG:teuthology.task.selinux:Excluding smithi203: OS 'ubuntu' does not support SELinux 2024-04-25T21:30:01.885 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T21:30:01.885 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T21:30:01.885 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T21:30:01.885 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T21:30:01.893 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T21:30:01.909 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T21:30:01.910 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T21:37:09.443 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-25T21:37:09.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-25T21:37:09.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:37:09.558 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-04-25T21:37:09.642 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-04-25T21:37:09.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T21:37:09.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T21:37:09.727 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-25T21:37:09.802 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T21:37:09.802 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T21:37:09.813 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T21:37:09.813 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T21:37:09.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T21:37:09.816 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T21:37:09.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T21:37:09.840 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Command line: ntpd -gq 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: ---------------------------------------------------- 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: corporation. Support and training for ntp-4 are 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: available at https://www.nwtime.org/support 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: ---------------------------------------------------- 2024-04-25T21:37:09.841 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: proto: precision = 0.048 usec (-24) 2024-04-25T21:37:09.842 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: basedate set to 2022-02-04 2024-04-25T21:37:09.842 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: gps base set to 2022-02-06 (week 2196) 2024-04-25T21:37:09.842 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T21:37:09.842 INFO:teuthology.orchestra.run.smithi006.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T21:37:09.842 INFO:teuthology.orchestra.run.smithi006.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T21:37:09.843 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T21:37:09.843 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T21:37:09.844 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T21:37:09.844 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T21:37:09.844 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Listen normally on 3 ens1f0 172.21.15.6:123 2024-04-25T21:37:09.844 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Listen normally on 4 lo [::1]:123 2024-04-25T21:37:09.844 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e4%4]:123 2024-04-25T21:37:09.844 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:09 ntpd[17102]: Listening on routing socket on fd #22 for interface updates 2024-04-25T21:37:09.871 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T21:37:09.871 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Command line: ntpd -gq 2024-04-25T21:37:09.872 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: ---------------------------------------------------- 2024-04-25T21:37:09.872 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T21:37:09.872 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T21:37:09.872 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: corporation. Support and training for ntp-4 are 2024-04-25T21:37:09.872 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: available at https://www.nwtime.org/support 2024-04-25T21:37:09.872 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: ---------------------------------------------------- 2024-04-25T21:37:09.873 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: proto: precision = 0.046 usec (-24) 2024-04-25T21:37:09.873 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: basedate set to 2022-02-04 2024-04-25T21:37:09.873 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: gps base set to 2022-02-06 (week 2196) 2024-04-25T21:37:09.873 INFO:teuthology.orchestra.run.smithi203.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T21:37:09.874 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T21:37:09.874 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T21:37:09.874 INFO:teuthology.orchestra.run.smithi203.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T21:37:09.875 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T21:37:09.875 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T21:37:09.875 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T21:37:09.875 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Listen normally on 3 enp3s0f1 172.21.15.203:123 2024-04-25T21:37:09.875 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Listen normally on 4 lo [::1]:123 2024-04-25T21:37:09.875 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4a23%5]:123 2024-04-25T21:37:09.875 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:09 ntpd[17200]: Listening on routing socket on fd #22 for interface updates 2024-04-25T21:37:10.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:10.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:10.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:10.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:10.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:10.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:10.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:10.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:10 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:11.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:11 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:11.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:11 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:11.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:11 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:11.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:11 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:11.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:11 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:11.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:11 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:12.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:12 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:12.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:12 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:12.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:12 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:12.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:12 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:12.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:12 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:12.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:12 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:13.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:13 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:13.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:13 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:13.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:13 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:13.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:13 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:13.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:13 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:13.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:13 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:13.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:13 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:13.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:13 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:13.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:13 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:13.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:13 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:13.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:13 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:13.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:13 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:14.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:14.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:14.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:14 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:14.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:14 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:15.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:15 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:15.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:15 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:15.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:15 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:15.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:15 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:15.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:15 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:15.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:15 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:15.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:15 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:15.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:15 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:15.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:15 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:15.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:15 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:15.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:15 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:15.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:15 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:16.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:16.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:16 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:16.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:16.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:16.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:16.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:16.872 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:16.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:16.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:16.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:16.873 INFO:teuthology.orchestra.run.smithi203.stdout:25 Apr 21:37:16 ntpd[17200]: ntpd: time slew -0.001054 s 2024-04-25T21:37:16.873 INFO:teuthology.orchestra.run.smithi203.stdout:ntpd: time slew -0.001054s 2024-04-25T21:37:16.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T21:37:16.873 INFO:teuthology.orchestra.run.smithi203.stderr:25 Apr 21:37:16 ntpd[17200]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T21:37:16.996 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T21:37:16.996 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-04-25T21:37:16.996 INFO:teuthology.orchestra.run.smithi203.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:16.996 INFO:teuthology.orchestra.run.smithi203.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:16.996 INFO:teuthology.orchestra.run.smithi203.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:16.996 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:17 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:17 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:17 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:17 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:17.841 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:17 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:17.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:17 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:18.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:18.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:18.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:18.843 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:18 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:19.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:19.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:19.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T21:37:19.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T21:37:19.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T21:37:19.842 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T21:37:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:25 Apr 21:37:19 ntpd[17102]: ntpd: time slew +0.001498 s 2024-04-25T21:37:19.843 INFO:teuthology.orchestra.run.smithi006.stdout:ntpd: time slew +0.001498s 2024-04-25T21:37:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T21:37:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:25 Apr 21:37:19 ntpd[17102]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T21:37:19.903 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T21:37:19.903 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-04-25T21:37:19.903 INFO:teuthology.orchestra.run.smithi006.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:19.903 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:19.903 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:19.903 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:37:19.904 INFO:teuthology.run_tasks:Running task install... 2024-04-25T21:37:19.915 DEBUG:teuthology.task.install:project ceph 2024-04-25T21:37:19.915 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T21:37:19.915 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T21:37:19.915 INFO:teuthology.task.install:Using flavor: default 2024-04-25T21:37:19.922 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T21:37:19.922 INFO:teuthology.task.install:extra packages: [] 2024-04-25T21:37:19.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-key list | grep Ceph 2024-04-25T21:37:19.923 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-key list | grep Ceph 2024-04-25T21:37:19.987 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T21:37:20.008 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T21:37:20.026 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T21:37:20.026 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T21:37:20.027 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-25T21:37:20.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T21:37:20.035 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T21:37:20.037 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T21:37:20.037 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-25T21:37:20.037 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T21:37:20.188 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-25T21:37:20.188 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:37:20.244 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-25T21:37:20.244 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:37:20.357 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:37:20.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T21:37:20.369 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:37:20.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T21:37:20.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-04-25T21:37:20.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-04-25T21:37:20.541 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T21:37:20.542 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T21:37:20.555 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T21:37:20.558 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T21:37:20.575 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T21:37:20.588 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T21:37:20.609 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T21:37:20.631 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T21:37:20.652 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-25T21:37:20.679 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T21:37:20.679 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-25T21:37:20.768 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T21:37:20.840 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T21:37:20.844 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T21:37:20.862 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T21:37:20.877 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 28.9 kB in 0s (88.0 kB/s) 2024-04-25T21:37:20.884 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T21:37:20.905 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-25T21:37:20.932 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-25T21:37:21.008 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 28.9 kB in 0s (65.8 kB/s) 2024-04-25T21:37:21.812 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-04-25T21:37:21.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:37:21.891 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-04-25T21:37:21.987 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-25T21:37:22.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:37:22.066 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-04-25T21:37:22.090 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-04-25T21:37:22.091 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-04-25T21:37:22.266 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-04-25T21:37:22.266 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-04-25T21:37:22.290 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T21:37:22.291 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T21:37:22.291 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T21:37:22.291 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T21:37:22.291 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T21:37:22.291 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T21:37:22.291 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T21:37:22.292 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-04-25T21:37:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T21:37:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T21:37:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T21:37:22.293 INFO:teuthology.orchestra.run.smithi203.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T21:37:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T21:37:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T21:37:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T21:37:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T21:37:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T21:37:22.295 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-25T21:37:22.296 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T21:37:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T21:37:22.297 INFO:teuthology.orchestra.run.smithi203.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T21:37:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T21:37:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-25T21:37:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet zip 2024-04-25T21:37:22.298 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-04-25T21:37:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T21:37:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T21:37:22.298 INFO:teuthology.orchestra.run.smithi203.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T21:37:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T21:37:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T21:37:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T21:37:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T21:37:22.299 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-04-25T21:37:22.299 INFO:teuthology.orchestra.run.smithi203.stdout: btrfs-tools 2024-04-25T21:37:22.397 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-04-25T21:37:22.397 INFO:teuthology.orchestra.run.smithi203.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T21:37:22.397 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T21:37:22.397 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T21:37:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-25T21:37:22.399 INFO:teuthology.orchestra.run.smithi203.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-25T21:37:22.463 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T21:37:22.464 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 179 MB of archives. 2024-04-25T21:37:22.464 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-25T21:37:22.464 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T21:37:22.468 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T21:37:22.469 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T21:37:22.469 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T21:37:22.469 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T21:37:22.469 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T21:37:22.469 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T21:37:22.469 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T21:37:22.470 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-04-25T21:37:22.470 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T21:37:22.470 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T21:37:22.470 INFO:teuthology.orchestra.run.smithi006.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T21:37:22.470 INFO:teuthology.orchestra.run.smithi006.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T21:37:22.471 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T21:37:22.471 INFO:teuthology.orchestra.run.smithi006.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T21:37:22.471 INFO:teuthology.orchestra.run.smithi006.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T21:37:22.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T21:37:22.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T21:37:22.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T21:37:22.471 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-25T21:37:22.472 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet zip 2024-04-25T21:37:22.473 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-04-25T21:37:22.473 INFO:teuthology.orchestra.run.smithi006.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T21:37:22.473 INFO:teuthology.orchestra.run.smithi006.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T21:37:22.473 INFO:teuthology.orchestra.run.smithi006.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T21:37:22.473 INFO:teuthology.orchestra.run.smithi006.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T21:37:22.473 INFO:teuthology.orchestra.run.smithi006.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T21:37:22.474 INFO:teuthology.orchestra.run.smithi006.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T21:37:22.474 INFO:teuthology.orchestra.run.smithi006.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T21:37:22.474 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-04-25T21:37:22.474 INFO:teuthology.orchestra.run.smithi006.stdout: btrfs-tools 2024-04-25T21:37:22.542 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-25T21:37:22.566 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T21:37:22.571 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T21:37:22.582 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-04-25T21:37:22.582 INFO:teuthology.orchestra.run.smithi006.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T21:37:22.582 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T21:37:22.582 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-25T21:37:22.583 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-25T21:37:22.583 INFO:teuthology.orchestra.run.smithi006.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-25T21:37:22.584 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-25T21:37:22.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-25T21:37:22.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-25T21:37:22.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-25T21:37:22.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-25T21:37:22.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-25T21:37:22.585 INFO:teuthology.orchestra.run.smithi006.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-25T21:37:22.589 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T21:37:22.653 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T21:37:22.653 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 179 MB of archives. 2024-04-25T21:37:22.653 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-25T21:37:22.653 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T21:37:22.656 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T21:37:22.661 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T21:37:22.675 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T21:37:22.679 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T21:37:22.679 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T21:37:22.680 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T21:37:22.688 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T21:37:22.690 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T21:37:22.692 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T21:37:22.693 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T21:37:22.704 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T21:37:22.708 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T21:37:22.712 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T21:37:22.715 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T21:37:22.718 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T21:37:22.723 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T21:37:22.725 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T21:37:22.730 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T21:37:22.735 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T21:37:22.738 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T21:37:22.743 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T21:37:22.744 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T21:37:22.744 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T21:37:22.747 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T21:37:22.749 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-25T21:37:22.762 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T21:37:22.763 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T21:37:22.763 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T21:37:22.764 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T21:37:22.771 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T21:37:22.773 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-25T21:37:22.774 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T21:37:22.782 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T21:37:22.782 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T21:37:22.784 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T21:37:22.785 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T21:37:22.785 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T21:37:22.790 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T21:37:22.796 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T21:37:22.802 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T21:37:22.802 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T21:37:22.803 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T21:37:22.804 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T21:37:22.822 INFO:teuthology.orchestra.run.smithi203.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T21:37:22.822 INFO:teuthology.orchestra.run.smithi203.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T21:37:22.824 INFO:teuthology.orchestra.run.smithi203.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T21:37:22.825 INFO:teuthology.orchestra.run.smithi203.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T21:37:22.827 INFO:teuthology.orchestra.run.smithi203.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T21:37:22.828 INFO:teuthology.orchestra.run.smithi203.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T21:37:22.842 INFO:teuthology.orchestra.run.smithi203.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T21:37:22.844 INFO:teuthology.orchestra.run.smithi203.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T21:37:22.880 INFO:teuthology.orchestra.run.smithi203.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-25T21:37:22.892 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T21:37:22.898 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T21:37:22.902 INFO:teuthology.orchestra.run.smithi203.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-25T21:37:22.914 INFO:teuthology.orchestra.run.smithi203.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-25T21:37:22.916 INFO:teuthology.orchestra.run.smithi203.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-25T21:37:22.927 INFO:teuthology.orchestra.run.smithi203.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-25T21:37:22.929 INFO:teuthology.orchestra.run.smithi203.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-25T21:37:22.929 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T21:37:22.934 INFO:teuthology.orchestra.run.smithi203.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T21:37:22.938 INFO:teuthology.orchestra.run.smithi203.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T21:37:22.938 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T21:37:22.941 INFO:teuthology.orchestra.run.smithi203.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-25T21:37:22.941 INFO:teuthology.orchestra.run.smithi203.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T21:37:22.941 INFO:teuthology.orchestra.run.smithi203.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T21:37:22.942 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T21:37:22.942 INFO:teuthology.orchestra.run.smithi203.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T21:37:22.943 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T21:37:23.016 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T21:37:23.020 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T21:37:23.022 INFO:teuthology.orchestra.run.smithi203.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T21:37:23.024 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T21:37:23.025 INFO:teuthology.orchestra.run.smithi203.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T21:37:23.025 INFO:teuthology.orchestra.run.smithi203.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T21:37:23.027 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T21:37:23.027 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T21:37:23.032 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T21:37:23.036 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T21:37:23.040 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T21:37:23.041 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T21:37:23.042 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T21:37:23.048 INFO:teuthology.orchestra.run.smithi203.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T21:37:23.048 INFO:teuthology.orchestra.run.smithi203.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T21:37:23.048 INFO:teuthology.orchestra.run.smithi203.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T21:37:23.049 INFO:teuthology.orchestra.run.smithi203.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T21:37:23.049 INFO:teuthology.orchestra.run.smithi203.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T21:37:23.050 INFO:teuthology.orchestra.run.smithi203.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T21:37:23.050 INFO:teuthology.orchestra.run.smithi203.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T21:37:23.062 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T21:37:23.063 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T21:37:23.064 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T21:37:23.064 INFO:teuthology.orchestra.run.smithi203.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T21:37:23.064 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T21:37:23.065 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T21:37:23.066 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T21:37:23.068 INFO:teuthology.orchestra.run.smithi203.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T21:37:23.083 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T21:37:23.083 INFO:teuthology.orchestra.run.smithi203.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T21:37:23.086 INFO:teuthology.orchestra.run.smithi203.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T21:37:23.087 INFO:teuthology.orchestra.run.smithi203.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T21:37:23.090 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T21:37:23.091 INFO:teuthology.orchestra.run.smithi203.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T21:37:23.092 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T21:37:23.092 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T21:37:23.094 INFO:teuthology.orchestra.run.smithi203.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T21:37:23.095 INFO:teuthology.orchestra.run.smithi203.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T21:37:23.096 INFO:teuthology.orchestra.run.smithi203.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T21:37:23.101 INFO:teuthology.orchestra.run.smithi203.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T21:37:23.105 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T21:37:23.106 INFO:teuthology.orchestra.run.smithi203.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T21:37:23.107 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T21:37:23.107 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T21:37:23.109 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T21:37:23.110 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T21:37:23.111 INFO:teuthology.orchestra.run.smithi203.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T21:37:23.112 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T21:37:23.126 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T21:37:23.126 INFO:teuthology.orchestra.run.smithi203.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T21:37:23.127 INFO:teuthology.orchestra.run.smithi203.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T21:37:23.128 INFO:teuthology.orchestra.run.smithi203.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T21:37:23.130 INFO:teuthology.orchestra.run.smithi203.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T21:37:23.132 INFO:teuthology.orchestra.run.smithi203.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T21:37:23.142 INFO:teuthology.orchestra.run.smithi203.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T21:37:23.142 INFO:teuthology.orchestra.run.smithi203.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T21:37:23.143 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T21:37:23.143 INFO:teuthology.orchestra.run.smithi203.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T21:37:23.144 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T21:37:23.145 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T21:37:23.148 INFO:teuthology.orchestra.run.smithi203.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T21:37:23.149 INFO:teuthology.orchestra.run.smithi203.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T21:37:23.149 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T21:37:23.152 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T21:37:23.158 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T21:37:23.160 INFO:teuthology.orchestra.run.smithi006.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T21:37:23.162 INFO:teuthology.orchestra.run.smithi006.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T21:37:23.165 INFO:teuthology.orchestra.run.smithi006.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T21:37:23.177 INFO:teuthology.orchestra.run.smithi006.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T21:37:23.178 INFO:teuthology.orchestra.run.smithi006.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T21:37:23.180 INFO:teuthology.orchestra.run.smithi006.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T21:37:23.185 INFO:teuthology.orchestra.run.smithi006.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T21:37:23.191 INFO:teuthology.orchestra.run.smithi203.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-25T21:37:23.193 INFO:teuthology.orchestra.run.smithi203.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-25T21:37:23.208 INFO:teuthology.orchestra.run.smithi203.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-25T21:37:23.258 INFO:teuthology.orchestra.run.smithi006.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T21:37:23.258 INFO:teuthology.orchestra.run.smithi006.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T21:37:23.259 INFO:teuthology.orchestra.run.smithi006.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T21:37:23.260 INFO:teuthology.orchestra.run.smithi006.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T21:37:23.261 INFO:teuthology.orchestra.run.smithi006.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T21:37:23.279 INFO:teuthology.orchestra.run.smithi006.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T21:37:23.281 INFO:teuthology.orchestra.run.smithi006.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T21:37:23.281 INFO:teuthology.orchestra.run.smithi006.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T21:37:23.296 INFO:teuthology.orchestra.run.smithi006.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T21:37:23.296 INFO:teuthology.orchestra.run.smithi006.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T21:37:23.302 INFO:teuthology.orchestra.run.smithi006.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T21:37:23.303 INFO:teuthology.orchestra.run.smithi006.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T21:37:23.304 INFO:teuthology.orchestra.run.smithi006.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T21:37:23.304 INFO:teuthology.orchestra.run.smithi006.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T21:37:23.304 INFO:teuthology.orchestra.run.smithi006.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T21:37:23.323 INFO:teuthology.orchestra.run.smithi006.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T21:37:23.328 INFO:teuthology.orchestra.run.smithi006.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T21:37:23.330 INFO:teuthology.orchestra.run.smithi006.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T21:37:23.332 INFO:teuthology.orchestra.run.smithi006.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T21:37:23.333 INFO:teuthology.orchestra.run.smithi006.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T21:37:23.344 INFO:teuthology.orchestra.run.smithi006.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T21:37:23.347 INFO:teuthology.orchestra.run.smithi006.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T21:37:23.348 INFO:teuthology.orchestra.run.smithi006.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T21:37:23.349 INFO:teuthology.orchestra.run.smithi006.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T21:37:23.354 INFO:teuthology.orchestra.run.smithi006.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T21:37:23.364 INFO:teuthology.orchestra.run.smithi006.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T21:37:23.368 INFO:teuthology.orchestra.run.smithi006.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T21:37:23.368 INFO:teuthology.orchestra.run.smithi006.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T21:37:23.369 INFO:teuthology.orchestra.run.smithi006.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T21:37:23.370 INFO:teuthology.orchestra.run.smithi006.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T21:37:23.385 INFO:teuthology.orchestra.run.smithi006.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T21:37:23.386 INFO:teuthology.orchestra.run.smithi006.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T21:37:23.394 INFO:teuthology.orchestra.run.smithi006.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T21:37:23.395 INFO:teuthology.orchestra.run.smithi006.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T21:37:23.395 INFO:teuthology.orchestra.run.smithi006.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T21:37:23.405 INFO:teuthology.orchestra.run.smithi006.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T21:37:23.406 INFO:teuthology.orchestra.run.smithi006.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T21:37:23.937 INFO:teuthology.orchestra.run.smithi006.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-25T21:37:24.252 INFO:teuthology.orchestra.run.smithi203.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-25T21:37:24.345 INFO:teuthology.orchestra.run.smithi203.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-25T21:37:24.351 INFO:teuthology.orchestra.run.smithi203.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-25T21:37:24.355 INFO:teuthology.orchestra.run.smithi203.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-25T21:37:24.413 INFO:teuthology.orchestra.run.smithi203.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-25T21:37:24.640 INFO:teuthology.orchestra.run.smithi203.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-25T21:37:24.857 INFO:teuthology.orchestra.run.smithi006.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-25T21:37:25.060 INFO:teuthology.orchestra.run.smithi006.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-25T21:37:25.132 INFO:teuthology.orchestra.run.smithi006.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-25T21:37:25.134 INFO:teuthology.orchestra.run.smithi006.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-25T21:37:25.201 INFO:teuthology.orchestra.run.smithi006.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-25T21:37:25.203 INFO:teuthology.orchestra.run.smithi006.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-25T21:37:25.337 INFO:teuthology.orchestra.run.smithi006.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-25T21:37:25.584 INFO:teuthology.orchestra.run.smithi203.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-25T21:37:25.584 INFO:teuthology.orchestra.run.smithi203.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-25T21:37:25.602 INFO:teuthology.orchestra.run.smithi203.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-25T21:37:25.644 INFO:teuthology.orchestra.run.smithi203.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-25T21:37:25.654 INFO:teuthology.orchestra.run.smithi203.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-25T21:37:25.656 INFO:teuthology.orchestra.run.smithi203.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-25T21:37:25.728 INFO:teuthology.orchestra.run.smithi203.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-25T21:37:26.050 INFO:teuthology.orchestra.run.smithi203.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-25T21:37:26.051 INFO:teuthology.orchestra.run.smithi203.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-25T21:37:27.247 INFO:teuthology.orchestra.run.smithi006.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-25T21:37:27.311 INFO:teuthology.orchestra.run.smithi006.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-25T21:37:27.423 INFO:teuthology.orchestra.run.smithi006.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-25T21:37:28.174 INFO:teuthology.orchestra.run.smithi203.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-25T21:37:28.176 INFO:teuthology.orchestra.run.smithi203.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-25T21:37:28.177 INFO:teuthology.orchestra.run.smithi203.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-25T21:37:28.607 INFO:teuthology.orchestra.run.smithi203.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-25T21:37:29.188 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 179 MB in 6s (28.9 MB/s) 2024-04-25T21:37:29.315 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T21:37:29.414 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T21:37:29.419 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T21:37:29.489 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T21:37:29.809 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T21:37:29.835 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T21:37:29.875 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T21:37:30.168 INFO:teuthology.orchestra.run.smithi006.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-25T21:37:30.212 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T21:37:30.238 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T21:37:30.278 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T21:37:30.481 INFO:teuthology.orchestra.run.smithi006.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-25T21:37:30.500 INFO:teuthology.orchestra.run.smithi006.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-25T21:37:30.503 INFO:teuthology.orchestra.run.smithi006.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-25T21:37:30.606 INFO:teuthology.orchestra.run.smithi006.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-25T21:37:30.682 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T21:37:30.708 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T21:37:30.756 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:37:31.016 INFO:teuthology.orchestra.run.smithi006.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-25T21:37:31.118 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T21:37:31.145 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T21:37:31.184 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:37:31.538 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T21:37:31.564 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T21:37:31.604 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:37:31.942 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T21:37:31.967 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T21:37:32.007 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T21:37:32.268 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librados2. 2024-04-25T21:37:32.290 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:32.334 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:32.478 INFO:teuthology.orchestra.run.smithi006.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-25T21:37:32.479 INFO:teuthology.orchestra.run.smithi006.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-25T21:37:32.501 INFO:teuthology.orchestra.run.smithi006.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-25T21:37:32.629 INFO:teuthology.orchestra.run.smithi006.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-25T21:37:32.670 INFO:teuthology.orchestra.run.smithi006.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-25T21:37:32.686 INFO:teuthology.orchestra.run.smithi006.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-25T21:37:32.809 INFO:teuthology.orchestra.run.smithi006.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-25T21:37:32.821 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librbd1. 2024-04-25T21:37:32.847 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:32.887 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:33.265 INFO:teuthology.orchestra.run.smithi006.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-25T21:37:33.265 INFO:teuthology.orchestra.run.smithi006.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-25T21:37:33.307 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T21:37:33.333 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:33.373 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:33.651 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rados. 2024-04-25T21:37:33.666 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:33.701 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:33.987 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T21:37:34.014 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:37:34.053 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:34.306 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T21:37:34.324 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:34.364 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:34.817 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T21:37:34.844 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:37:34.883 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:35.228 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T21:37:35.254 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T21:37:35.311 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T21:37:35.598 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T21:37:35.625 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T21:37:35.656 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T21:37:35.891 INFO:teuthology.orchestra.run.smithi006.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-25T21:37:35.894 INFO:teuthology.orchestra.run.smithi006.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-25T21:37:35.895 INFO:teuthology.orchestra.run.smithi006.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-25T21:37:35.917 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T21:37:35.943 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:35.983 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:36.336 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T21:37:36.363 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T21:37:36.403 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T21:37:36.441 INFO:teuthology.orchestra.run.smithi006.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-25T21:37:36.647 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T21:37:36.662 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T21:37:36.705 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T21:37:37.008 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T21:37:37.023 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T21:37:37.027 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 179 MB in 14s (13.0 MB/s) 2024-04-25T21:37:37.066 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T21:37:37.179 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T21:37:37.236 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T21:37:37.241 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T21:37:37.327 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T21:37:37.378 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua5.1. 2024-04-25T21:37:37.404 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T21:37:37.444 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T21:37:37.672 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T21:37:37.699 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T21:37:37.739 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T21:37:37.739 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-any. 2024-04-25T21:37:37.754 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-25T21:37:37.789 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T21:37:38.033 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package zip. 2024-04-25T21:37:38.060 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-25T21:37:38.092 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T21:37:38.100 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T21:37:38.118 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T21:37:38.175 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T21:37:38.411 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package unzip. 2024-04-25T21:37:38.438 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T21:37:38.477 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T21:37:38.537 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T21:37:38.564 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T21:37:38.603 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:37:38.859 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package luarocks. 2024-04-25T21:37:38.885 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T21:37:38.925 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T21:37:39.023 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T21:37:39.050 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T21:37:39.090 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:37:39.301 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librgw2. 2024-04-25T21:37:39.327 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:39.367 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:39.443 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T21:37:39.472 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T21:37:39.510 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:37:39.813 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T21:37:39.839 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T21:37:39.871 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T21:37:39.903 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T21:37:39.918 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:39.961 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:40.208 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librados2. 2024-04-25T21:37:40.234 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:40.274 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:40.281 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T21:37:40.308 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T21:37:40.339 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T21:37:40.626 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T21:37:40.653 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:40.692 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:40.754 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librbd1. 2024-04-25T21:37:40.778 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:40.818 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:41.121 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-common. 2024-04-25T21:37:41.148 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:41.187 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:41.262 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T21:37:41.289 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:41.329 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:41.615 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rados. 2024-04-25T21:37:41.630 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:41.673 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:42.010 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T21:37:42.036 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:37:42.076 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:42.329 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T21:37:42.355 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:42.395 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:42.535 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-base. 2024-04-25T21:37:42.562 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:42.611 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:42.664 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T21:37:42.680 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:37:42.714 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:43.018 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T21:37:43.045 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T21:37:43.084 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T21:37:43.190 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T21:37:43.216 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T21:37:43.256 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T21:37:43.403 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T21:37:43.420 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T21:37:43.462 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T21:37:43.576 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T21:37:43.602 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T21:37:43.643 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T21:37:43.722 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T21:37:43.738 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:43.781 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:43.984 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T21:37:44.010 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T21:37:44.045 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T21:37:44.101 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T21:37:44.127 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T21:37:44.167 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T21:37:44.340 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T21:37:44.367 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T21:37:44.407 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T21:37:44.411 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T21:37:44.427 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T21:37:44.469 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T21:37:44.710 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T21:37:44.736 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T21:37:44.776 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T21:37:44.789 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T21:37:44.805 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T21:37:44.847 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T21:37:45.054 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T21:37:45.080 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T21:37:45.121 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T21:37:45.201 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua5.1. 2024-04-25T21:37:45.227 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T21:37:45.267 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T21:37:45.416 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-portend. 2024-04-25T21:37:45.442 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-25T21:37:45.482 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T21:37:45.570 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-any. 2024-04-25T21:37:45.586 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-25T21:37:45.620 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T21:37:45.785 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T21:37:45.812 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T21:37:45.852 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T21:37:45.856 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package zip. 2024-04-25T21:37:45.882 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-25T21:37:45.922 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T21:37:46.155 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T21:37:46.181 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T21:37:46.222 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T21:37:46.242 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package unzip. 2024-04-25T21:37:46.269 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T21:37:46.310 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T21:37:46.568 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T21:37:46.594 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T21:37:46.634 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T21:37:46.704 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package luarocks. 2024-04-25T21:37:46.731 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T21:37:46.771 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T21:37:46.946 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T21:37:46.972 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T21:37:47.012 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T21:37:47.120 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librgw2. 2024-04-25T21:37:47.147 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:47.178 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:47.290 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T21:37:47.317 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T21:37:47.357 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T21:37:47.660 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-mako. 2024-04-25T21:37:47.686 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T21:37:47.714 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T21:37:47.726 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T21:37:47.741 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:47.781 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:48.088 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T21:37:48.109 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T21:37:48.114 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T21:37:48.137 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T21:37:48.155 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T21:37:48.359 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T21:37:48.442 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T21:37:48.468 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T21:37:48.508 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T21:37:48.678 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T21:37:48.705 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:48.745 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webob. 2024-04-25T21:37:48.837 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T21:37:48.877 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T21:37:49.115 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-common. 2024-04-25T21:37:49.141 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:49.181 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:49.197 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T21:37:49.224 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T21:37:49.259 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T21:37:49.542 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T21:37:49.569 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T21:37:49.609 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T21:37:49.912 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-paste. 2024-04-25T21:37:49.939 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T21:37:49.978 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T21:37:50.315 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T21:37:50.330 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T21:37:50.373 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T21:37:50.553 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-base. 2024-04-25T21:37:50.580 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:50.629 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:50.676 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T21:37:50.703 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T21:37:50.743 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T21:37:51.030 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T21:37:51.056 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T21:37:51.096 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T21:37:51.191 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T21:37:51.217 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T21:37:51.257 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T21:37:51.427 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T21:37:51.451 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T21:37:51.483 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T21:37:51.535 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T21:37:51.562 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T21:37:51.602 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T21:37:51.846 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T21:37:51.873 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T21:37:51.904 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T21:37:51.929 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T21:37:51.956 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T21:37:51.996 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T21:37:52.233 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-idna. 2024-04-25T21:37:52.259 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-25T21:37:52.299 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T21:37:52.307 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T21:37:52.334 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T21:37:52.365 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T21:37:52.610 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T21:37:52.637 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T21:37:52.662 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T21:37:52.677 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T21:37:52.687 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T21:37:52.726 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T21:37:53.005 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests. 2024-04-25T21:37:53.029 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T21:37:53.032 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T21:37:53.056 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T21:37:53.072 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T21:37:53.096 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T21:37:53.367 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T21:37:53.383 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T21:37:53.398 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-portend. 2024-04-25T21:37:53.417 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T21:37:53.425 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-25T21:37:53.465 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T21:37:53.703 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T21:37:53.709 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T21:37:53.719 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:37:53.725 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T21:37:53.753 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:53.768 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T21:37:54.071 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T21:37:54.098 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T21:37:54.109 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T21:37:54.136 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:54.137 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T21:37:54.167 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:54.449 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T21:37:54.462 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T21:37:54.465 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T21:37:54.488 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T21:37:54.499 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T21:37:54.528 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T21:37:54.786 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T21:37:54.813 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T21:37:54.814 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T21:37:54.841 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T21:37:54.852 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T21:37:54.881 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T21:37:55.155 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T21:37:55.182 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T21:37:55.184 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T21:37:55.199 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:55.221 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T21:37:55.234 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:55.508 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-mako. 2024-04-25T21:37:55.534 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T21:37:55.574 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T21:37:55.621 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T21:37:55.648 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:55.687 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:56.308 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T21:37:56.335 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:56.374 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T21:37:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T21:37:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T21:37:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T21:37:56.949 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T21:37:56.988 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T21:37:57.291 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webob. 2024-04-25T21:37:57.318 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T21:37:57.358 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T21:37:57.403 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph. 2024-04-25T21:37:57.418 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:57.453 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:57.719 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T21:37:57.730 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T21:37:57.746 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T21:37:57.757 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:57.789 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T21:37:57.797 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:58.105 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T21:37:58.132 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T21:37:58.142 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T21:37:58.169 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:58.172 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T21:37:58.209 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:58.433 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-paste. 2024-04-25T21:37:58.460 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T21:37:58.491 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T21:37:58.603 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package cephadm. 2024-04-25T21:37:58.619 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:37:58.653 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:37:58.811 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T21:37:58.827 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T21:37:58.869 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T21:37:58.948 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T21:37:58.975 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T21:37:59.014 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T21:37:59.130 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T21:37:59.157 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T21:37:59.189 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T21:37:59.351 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T21:37:59.367 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T21:37:59.401 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T21:37:59.467 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T21:37:59.494 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T21:37:59.533 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T21:37:59.853 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T21:37:59.880 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T21:37:59.919 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T21:38:00.316 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T21:38:00.343 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T21:38:00.382 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T21:38:00.689 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-babel. 2024-04-25T21:38:00.716 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T21:38:00.718 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-idna. 2024-04-25T21:38:00.746 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-25T21:38:00.755 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T21:38:00.785 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T21:38:01.076 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T21:38:01.079 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T21:38:01.102 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T21:38:01.106 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T21:38:01.138 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T21:38:01.142 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T21:38:01.433 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests. 2024-04-25T21:38:01.459 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T21:38:01.516 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T21:38:01.521 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T21:38:01.547 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:01.587 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:01.794 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T21:38:01.809 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T21:38:01.852 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T21:38:01.924 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T21:38:01.951 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T21:38:01.991 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T21:38:02.138 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T21:38:02.154 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:02.188 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:02.294 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-routes. 2024-04-25T21:38:02.321 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T21:38:02.360 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T21:38:02.568 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T21:38:02.596 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:02.634 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:02.689 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T21:38:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:02.755 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:02.921 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T21:38:02.948 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T21:38:02.987 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T21:38:03.290 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T21:38:03.317 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T21:38:03.356 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T21:38:03.684 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T21:38:03.701 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:03.751 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:04.045 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T21:38:04.072 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T21:38:04.111 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T21:38:04.188 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T21:38:04.215 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:04.254 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:04.598 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T21:38:04.625 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T21:38:04.665 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T21:38:04.857 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T21:38:04.884 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:04.924 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:05.027 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T21:38:05.054 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T21:38:05.093 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T21:38:05.405 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T21:38:05.432 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T21:38:05.471 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T21:38:06.027 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph. 2024-04-25T21:38:06.044 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:06.085 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:06.197 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T21:38:06.225 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:06.346 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T21:38:06.349 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:06.374 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:06.404 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:06.774 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T21:38:06.801 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:06.841 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:07.244 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package cephadm. 2024-04-25T21:38:07.271 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:07.310 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:07.369 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T21:38:07.396 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T21:38:07.436 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T21:38:07.621 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T21:38:07.649 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T21:38:07.688 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T21:38:07.714 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T21:38:07.741 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T21:38:07.772 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T21:38:08.040 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T21:38:08.057 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T21:38:08.067 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T21:38:08.083 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T21:38:08.099 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T21:38:08.125 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T21:38:08.404 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T21:38:08.431 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-25T21:38:08.470 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T21:38:08.791 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T21:38:08.818 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T21:38:08.857 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T21:38:09.137 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-babel. 2024-04-25T21:38:09.154 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T21:38:09.195 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T21:38:09.203 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T21:38:09.218 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T21:38:09.261 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T21:38:09.515 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T21:38:09.542 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T21:38:09.572 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T21:38:09.581 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T21:38:09.599 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T21:38:09.639 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T21:38:09.918 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T21:38:09.945 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:09.967 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T21:38:09.993 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:09.995 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T21:38:10.034 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T21:38:10.430 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T21:38:10.457 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T21:38:10.496 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T21:38:10.682 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T21:38:10.709 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:10.740 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-routes. 2024-04-25T21:38:10.748 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:10.768 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T21:38:10.806 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T21:38:11.118 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T21:38:11.144 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T21:38:11.145 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T21:38:11.171 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:11.184 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T21:38:11.210 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:11.521 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T21:38:11.548 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T21:38:11.587 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T21:38:11.832 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package jq. 2024-04-25T21:38:11.848 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T21:38:11.890 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T21:38:12.151 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package socat. 2024-04-25T21:38:12.179 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T21:38:12.218 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T21:38:12.515 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T21:38:12.530 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T21:38:12.543 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T21:38:12.557 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T21:38:12.581 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T21:38:12.596 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T21:38:12.942 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-test. 2024-04-25T21:38:12.958 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:13.001 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:13.085 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T21:38:13.112 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T21:38:13.151 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T21:38:13.530 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T21:38:13.557 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T21:38:13.597 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T21:38:13.874 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T21:38:13.901 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T21:38:13.932 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T21:38:14.726 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T21:38:14.754 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:14.886 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:15.314 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T21:38:15.341 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:15.380 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:15.678 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T21:38:15.705 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:15.744 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:15.923 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T21:38:15.950 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T21:38:15.989 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T21:38:16.250 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T21:38:16.266 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T21:38:16.289 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T21:38:16.300 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T21:38:16.316 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T21:38:16.347 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T21:38:16.570 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T21:38:16.586 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T21:38:16.628 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T21:38:16.661 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T21:38:16.688 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T21:38:16.727 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T21:38:16.915 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T21:38:16.942 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-25T21:38:16.981 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T21:38:17.098 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T21:38:17.125 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T21:38:17.165 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T21:38:17.301 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T21:38:17.329 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T21:38:17.368 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T21:38:17.595 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package pkg-config. 2024-04-25T21:38:17.623 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T21:38:17.662 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T21:38:17.688 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T21:38:17.715 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T21:38:17.754 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T21:38:17.898 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T21:38:17.925 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T21:38:17.965 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T21:38:18.058 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T21:38:18.085 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T21:38:18.115 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T21:38:18.360 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T21:38:18.387 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T21:38:18.418 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T21:38:18.435 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T21:38:18.463 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T21:38:18.502 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T21:38:18.754 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T21:38:18.782 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T21:38:18.821 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T21:38:19.141 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T21:38:19.169 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T21:38:19.207 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T21:38:19.449 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T21:38:19.478 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:19.515 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:19.519 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T21:38:19.546 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T21:38:19.585 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T21:38:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T21:38:19.888 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T21:38:19.926 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T21:38:19.930 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-py. 2024-04-25T21:38:19.958 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-25T21:38:20.005 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T21:38:20.280 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T21:38:20.308 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T21:38:20.326 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T21:38:20.346 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T21:38:20.353 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T21:38:20.392 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T21:38:20.574 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package jq. 2024-04-25T21:38:20.602 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T21:38:20.632 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T21:38:20.846 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T21:38:20.868 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package socat. 2024-04-25T21:38:20.874 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T21:38:20.884 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T21:38:20.913 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T21:38:20.926 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T21:38:21.224 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-toml. 2024-04-25T21:38:21.252 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-25T21:38:21.254 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T21:38:21.271 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T21:38:21.290 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T21:38:21.321 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T21:38:21.593 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T21:38:21.621 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T21:38:21.642 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-test. 2024-04-25T21:38:21.660 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T21:38:21.670 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:21.708 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:22.006 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T21:38:22.033 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T21:38:22.072 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T21:38:22.342 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T21:38:22.369 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T21:38:22.416 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T21:38:23.146 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package radosgw. 2024-04-25T21:38:23.174 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:23.212 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:23.996 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T21:38:24.024 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T21:38:24.024 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-25T21:38:24.039 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:24.071 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:24.073 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:24.370 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T21:38:24.391 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T21:38:24.419 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:24.458 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:24.621 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T21:38:24.747 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T21:38:24.777 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T21:38:24.805 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T21:38:24.843 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T21:38:25.008 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T21:38:25.198 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T21:38:25.226 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T21:38:25.264 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T21:38:25.342 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T21:38:25.602 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T21:38:25.630 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T21:38:25.668 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T21:38:25.791 INFO:teuthology.orchestra.run.smithi203.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T21:38:25.899 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:26.057 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package pkg-config. 2024-04-25T21:38:26.085 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T21:38:26.123 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T21:38:26.384 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T21:38:26.401 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T21:38:26.442 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T21:38:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user cephadm....done 2024-04-25T21:38:26.723 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T21:38:26.870 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T21:38:26.898 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T21:38:26.937 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T21:38:27.193 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T21:38:27.231 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T21:38:27.260 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T21:38:27.322 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T21:38:27.444 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T21:38:27.570 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T21:38:27.651 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T21:38:27.679 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T21:38:27.717 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T21:38:27.821 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T21:38:27.970 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T21:38:27.987 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T21:38:28.020 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T21:38:28.073 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T21:38:28.174 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T21:38:28.356 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-py. 2024-04-25T21:38:28.384 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-25T21:38:28.414 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T21:38:28.475 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T21:38:28.701 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T21:38:28.717 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T21:38:28.751 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T21:38:28.819 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T21:38:29.079 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T21:38:29.163 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T21:38:29.191 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T21:38:29.230 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T21:38:29.346 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T21:38:29.549 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-toml. 2024-04-25T21:38:29.577 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-25T21:38:29.615 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T21:38:29.640 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T21:38:29.891 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T21:38:29.918 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T21:38:29.946 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T21:38:29.985 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T21:38:30.159 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T21:38:30.371 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T21:38:30.399 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T21:38:30.410 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:30.438 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T21:38:30.670 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T21:38:30.707 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T21:38:30.724 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T21:38:30.765 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T21:38:30.796 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T21:38:30.930 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T21:38:31.031 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T21:38:31.266 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T21:38:31.578 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package radosgw. 2024-04-25T21:38:31.606 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:31.644 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:31.835 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T21:38:32.039 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T21:38:32.111 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T21:38:32.237 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T21:38:32.481 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T21:38:32.508 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-25T21:38:32.547 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:32.605 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T21:38:32.706 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T21:38:32.852 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T21:38:32.949 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T21:38:33.152 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T21:38:33.208 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T21:38:33.261 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T21:38:33.510 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T21:38:33.528 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T21:38:33.636 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T21:38:33.851 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T21:38:34.300 INFO:teuthology.orchestra.run.smithi006.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T21:38:34.321 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T21:38:34.371 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:34.564 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T21:38:34.674 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T21:38:34.800 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T21:38:34.888 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user cephadm....done 2024-04-25T21:38:35.044 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T21:38:35.092 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T21:38:35.370 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T21:38:35.486 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T21:38:35.638 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T21:38:35.745 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T21:38:35.764 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T21:38:36.015 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T21:38:36.130 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T21:38:36.300 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T21:38:36.406 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T21:38:36.426 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T21:38:36.733 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T21:38:36.736 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T21:38:36.984 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T21:38:37.111 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:37.121 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T21:38:37.381 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T21:38:37.420 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T21:38:37.649 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T21:38:37.755 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T21:38:37.883 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T21:38:38.161 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T21:38:38.427 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T21:38:38.712 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:38.836 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T21:38:38.971 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T21:38:39.088 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:38:39.139 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T21:38:39.214 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T21:38:39.265 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T21:38:39.399 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T21:38:39.457 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T21:38:39.582 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T21:38:39.675 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T21:38:39.709 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T21:38:40.002 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T21:38:40.042 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T21:38:40.107 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T21:38:40.170 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T21:38:40.253 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T21:38:40.296 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T21:38:40.504 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:38:40.614 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T21:38:40.673 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T21:38:40.773 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T21:38:40.890 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T21:38:41.033 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T21:38:41.133 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T21:38:41.317 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T21:38:41.393 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T21:38:41.635 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T21:38:41.661 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T21:38:41.786 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T21:38:41.920 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T21:38:42.180 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T21:38:42.297 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T21:38:42.497 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T21:38:42.574 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T21:38:42.683 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T21:38:42.740 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T21:38:42.840 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T21:38:42.984 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T21:38:43.000 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T21:38:43.309 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T21:38:43.310 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T21:38:43.570 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T21:38:43.696 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T21:38:43.744 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T21:38:43.831 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T21:38:43.906 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T21:38:44.028 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T21:38:44.132 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T21:38:44.258 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T21:38:44.446 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T21:38:44.635 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T21:38:44.748 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T21:38:44.903 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T21:38:45.029 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T21:38:45.107 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T21:38:45.280 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:38:45.382 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T21:38:45.392 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T21:38:45.518 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:45.725 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T21:38:45.836 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T21:38:46.034 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T21:38:46.145 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T21:38:46.311 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T21:38:46.453 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T21:38:46.721 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T21:38:46.981 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T21:38:47.301 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T21:38:47.326 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T21:38:47.577 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:38:47.686 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T21:38:47.694 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T21:38:47.937 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T21:38:47.979 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T21:38:48.055 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T21:38:48.181 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T21:38:48.298 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T21:38:48.425 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:48.491 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T21:38:48.551 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:48.677 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T21:38:48.775 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T21:38:49.014 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:49.043 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:38:49.123 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:49.169 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T21:38:49.258 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T21:38:49.454 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T21:38:49.534 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:49.660 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:49.713 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T21:38:49.795 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:50.008 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T21:38:50.008 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T21:38:50.181 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T21:38:50.465 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T21:38:50.505 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:50.622 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:50.723 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:50.725 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T21:38:50.850 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T21:38:50.992 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T21:38:51.143 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T21:38:51.419 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T21:38:51.520 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T21:38:51.813 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T21:38:52.003 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:52.113 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:52.114 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T21:38:52.214 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:52.315 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:52.374 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T21:38:52.433 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:52.500 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T21:38:52.626 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T21:38:52.715 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T21:38:52.969 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T21:38:52.993 INFO:teuthology.orchestra.run.smithi203.stdout:Adding group ceph....done 2024-04-25T21:38:53.070 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T21:38:53.329 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user ceph....done 2024-04-25T21:38:53.447 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T21:38:53.463 INFO:teuthology.orchestra.run.smithi203.stdout:Setting system user ceph properties....done 2024-04-25T21:38:53.481 INFO:teuthology.orchestra.run.smithi203.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T21:38:53.619 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T21:38:53.715 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T21:38:53.840 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T21:38:53.928 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T21:38:54.108 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T21:38:54.234 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T21:38:54.435 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:54.560 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T21:38:54.561 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:54.861 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T21:38:55.025 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T21:38:55.025 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T21:38:55.121 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T21:38:55.238 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T21:38:55.505 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:55.531 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T21:38:55.807 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T21:38:55.843 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T21:38:56.169 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T21:38:56.340 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:56.545 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T21:38:56.552 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T21:38:56.552 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T21:38:56.814 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T21:38:57.052 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:57.158 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T21:38:57.284 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:57.367 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T21:38:57.367 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T21:38:57.410 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:57.527 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T21:38:57.822 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:57.849 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:57.948 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:58.074 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T21:38:58.227 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T21:38:58.228 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T21:38:58.359 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:58.485 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:58.603 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:58.708 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:58.808 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T21:38:58.809 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T21:38:58.834 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:58.977 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:59.199 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T21:38:59.199 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T21:38:59.329 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:59.438 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:59.573 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:59.699 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T21:38:59.711 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:59.846 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:38:59.964 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:00.115 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:00.460 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T21:39:00.586 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T21:39:00.731 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T21:39:00.935 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:01.036 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:01.137 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:01.247 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:01.381 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:01.515 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T21:39:01.907 INFO:teuthology.orchestra.run.smithi006.stdout:Adding group ceph....done 2024-04-25T21:39:02.186 INFO:teuthology.orchestra.run.smithi203.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T21:39:02.192 DEBUG:teuthology.parallel:result is None 2024-04-25T21:39:02.284 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user ceph....done 2024-04-25T21:39:02.438 INFO:teuthology.orchestra.run.smithi006.stdout:Setting system user ceph properties....done 2024-04-25T21:39:02.457 INFO:teuthology.orchestra.run.smithi006.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T21:39:02.599 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T21:39:02.892 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T21:39:03.376 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:03.502 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:03.943 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T21:39:03.943 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T21:39:04.429 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:04.796 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T21:39:05.382 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:05.751 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T21:39:05.752 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T21:39:06.237 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:06.526 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T21:39:06.526 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T21:39:07.046 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:07.330 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T21:39:07.330 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T21:39:07.856 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:07.991 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:08.142 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:08.368 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T21:39:08.368 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T21:39:08.826 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:08.961 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:09.204 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:09.330 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-25T21:39:09.674 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T21:39:09.809 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T21:39:09.961 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T21:39:11.290 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T21:39:11.953 INFO:teuthology.orchestra.run.smithi006.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T21:39:11.958 DEBUG:teuthology.parallel:result is None 2024-04-25T21:39:11.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T21:39:12.174 DEBUG:teuthology.orchestra.run.smithi006:> dpkg-query -W -f '${Version}' ceph 2024-04-25T21:39:12.194 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:39:12.195 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:39:12.195 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-25T21:39:12.196 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T21:39:12.335 DEBUG:teuthology.orchestra.run.smithi203:> dpkg-query -W -f '${Version}' ceph 2024-04-25T21:39:12.355 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:39:12.355 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-25T21:39:12.355 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-25T21:39:12.357 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T21:39:12.357 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:12.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T21:39:12.372 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:12.372 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T21:39:12.410 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T21:39:12.411 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:12.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T21:39:12.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T21:39:12.486 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:12.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T21:39:12.499 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T21:39:12.553 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T21:39:12.553 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:12.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T21:39:12.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T21:39:12.622 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:12.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T21:39:12.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T21:39:12.689 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T21:39:12.689 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:12.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T21:39:12.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T21:39:12.757 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:12.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T21:39:12.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T21:39:12.825 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T21:39:12.909 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T21:39:12.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-04-25T21:39:12.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-04-25T21:39:12.924 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T21:39:12.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T21:39:12.967 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T21:39:12.979 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T21:39:12.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T21:39:13.027 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T21:39:13.048 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T21:39:13.048 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-04-25T21:39:13.049 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740', 'branch': 'squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7673740', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'mon_bind_msgr2': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9886, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '30270/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNpKTVp25AOOPX/x1Pz8O7K0xHFoi+nouZxvb7b3qMPpNR7a0C6/1gczgfuh9x2uueTMajRol4aM3+Z25kf9EIw=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbLYmhHZNxHxKgC5WNhsun6mgkS/KmRQ1LXeyu+MFWgIWD0+lCQYDt6lgDIaouUANlA6ugDPBixcdaxOdWf99s='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_21:00:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T21:39:13.049 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T21:39:13.083 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T21:39:13.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T21:39:13.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T21:39:13.144 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:13.144 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T21:39:13.190 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T21:39:13.190 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-04-25T21:39:13.238 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T21:39:13.238 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:13.239 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 868 Links: 1 2024-04-25T21:39:13.239 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:13.239 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-25 21:33:48.148529527 +0000 2024-04-25T21:39:13.241 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-25 21:33:47.892531814 +0000 2024-04-25T21:39:13.241 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-25 21:33:47.892531814 +0000 2024-04-25T21:39:13.241 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-25T21:39:13.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T21:39:13.292 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-25T21:39:13.293 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-25T21:39:13.293 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000393505 s, 1.3 MB/s 2024-04-25T21:39:13.294 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T21:39:13.343 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-25 21:33:48.636525167 +0000 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-25 21:33:48.384527419 +0000 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-25 21:33:48.384527419 +0000 2024-04-25T21:39:13.390 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-25T21:39:13.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T21:39:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-25T21:39:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-25T21:39:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000410291 s, 1.2 MB/s 2024-04-25T21:39:13.445 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T21:39:13.495 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-25 21:33:49.128520773 +0000 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-25 21:33:48.876523024 +0000 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-25 21:33:48.876523024 +0000 2024-04-25T21:39:13.542 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-25T21:39:13.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T21:39:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-25T21:39:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-25T21:39:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000285906 s, 1.8 MB/s 2024-04-25T21:39:13.599 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T21:39:13.647 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-04-25T21:39:13.694 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T21:39:13.694 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:13.694 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-04-25T21:39:13.694 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:13.694 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-04-25 21:33:49.648516129 +0000 2024-04-25T21:39:13.694 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-04-25 21:33:49.356518737 +0000 2024-04-25T21:39:13.695 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-04-25 21:33:49.356518737 +0000 2024-04-25T21:39:13.695 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-04-25T21:39:13.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T21:39:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-04-25T21:39:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-04-25T21:39:13.748 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000378117 s, 1.4 MB/s 2024-04-25T21:39:13.749 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T21:39:13.799 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:13.799 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:13.799 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T21:39:13.805 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T21:39:13.805 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 880 Links: 1 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 21:33:48.303616936 +0000 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 21:33:48.027619402 +0000 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 21:33:48.027619402 +0000 2024-04-25T21:39:13.851 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-04-25T21:39:13.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T21:39:13.905 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T21:39:13.905 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T21:39:13.905 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.00044686 s, 1.1 MB/s 2024-04-25T21:39:13.907 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T21:39:13.956 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-25T21:39:14.003 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T21:39:14.003 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:14.003 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 897 Links: 1 2024-04-25T21:39:14.003 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:14.003 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 21:33:48.771612756 +0000 2024-04-25T21:39:14.004 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 21:33:48.523614971 +0000 2024-04-25T21:39:14.004 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 21:33:48.523614971 +0000 2024-04-25T21:39:14.004 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-04-25T21:39:14.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T21:39:14.057 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T21:39:14.057 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T21:39:14.057 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000420134 s, 1.2 MB/s 2024-04-25T21:39:14.058 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T21:39:14.107 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-25T21:39:14.154 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T21:39:14.155 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:14.155 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-04-25T21:39:14.155 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:14.155 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 21:33:49.003610683 +0000 2024-04-25T21:39:14.155 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 21:33:48.999610719 +0000 2024-04-25T21:39:14.155 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 21:33:48.999610719 +0000 2024-04-25T21:39:14.155 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-04-25T21:39:14.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T21:39:14.208 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T21:39:14.209 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T21:39:14.209 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.0004133 s, 1.2 MB/s 2024-04-25T21:39:14.210 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T21:39:14.260 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-25T21:39:14.306 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T21:39:14.306 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T21:39:14.307 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 929 Links: 1 2024-04-25T21:39:14.307 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T21:39:14.307 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 21:33:49.459606610 +0000 2024-04-25T21:39:14.307 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 21:33:49.455606646 +0000 2024-04-25T21:39:14.307 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 21:33:49.455606646 +0000 2024-04-25T21:39:14.307 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-04-25T21:39:14.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T21:39:14.361 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T21:39:14.361 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T21:39:14.361 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000380507 s, 1.3 MB/s 2024-04-25T21:39:14.363 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T21:39:14.412 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:14.412 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-25T21:39:14.412 INFO:tasks.ceph:Generating config... 2024-04-25T21:39:14.413 INFO:tasks.ceph:[client] debug monc = 20 2024-04-25T21:39:14.413 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T21:39:14.413 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-25T21:39:14.413 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-04-25T21:39:14.413 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-04-25T21:39:14.413 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-04-25T21:39:14.413 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-04-25T21:39:14.413 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-04-25T21:39:14.413 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-04-25T21:39:14.414 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-25T21:39:14.414 INFO:tasks.ceph:[global] ms type = async 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-04-25T21:39:14.414 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-04-25T21:39:14.415 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-25T21:39:14.415 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T21:39:14.415 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-25T21:39:14.416 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-25T21:39:14.416 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T21:39:14.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T21:39:14.452 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T21:39:14.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T21:39:14.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T21:39:14.606 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.6:6789]'), ('mon.c', '[v1:172.21.15.6:6790]'), ('mon.b', '[v1:172.21.15.203:6789]')] 2024-04-25T21:39:14.606 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.203:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-25T21:39:14.607 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:14.607 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T21:39:14.654 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.6:6789]' --addv c '[v1:172.21.15.6:6790]' --addv b '[v1:172.21.15.203:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-04-25T21:39:14.742 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 3 2024-04-25T21:39:14.742 INFO:teuthology.orchestra.run.smithi006.stdout:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:14.742 INFO:teuthology.orchestra.run.smithi006.stdout:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:14.742 INFO:teuthology.orchestra.run.smithi006.stdout:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:14.742 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T21:39:14.743 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T21:39:14.790 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7c01789f-5916-4cc5-8455-2e5d8f8fad2b... 2024-04-25T21:39:14.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T21:39:14.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.864 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-04-25T21:39:14.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T21:39:14.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.203:6789]" 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: mon election default strategy = 3 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: ms bind msgr2 = False 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay max = 1 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay probability = 0.005 2024-04-25T21:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay type = mon 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject internal delays = 0.002 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 2500 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: ms type = async 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-04-25T21:39:14.868 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.869 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore zero block detection = True 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-04-25T21:39:14.870 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify cached snaps = True 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify missing on start = True 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock override recovery settings = True 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd memory target = 1610612736 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue = debug_random 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue cut off = debug_random 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-04-25T21:39:14.871 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 10 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-04-25T21:39:14.872 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.873 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon min osdmap epochs = 25 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune interval = 2 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune min = 15 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune txsize = 2 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub inject crc mismatch = 0.01 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub inject missing keys = 0.05 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub interval = 300 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: paxos service trim min = 5 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-04-25T21:39:14.874 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-04-25T21:39:14.875 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-04-25T21:39:14.876 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-04-25T21:39:14.877 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-04-25T21:39:14.878 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-04-25T21:39:14.879 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "[v1:172.21.15.6:6789],[v1:172.21.15.6:6790],[v1:172.21.15.203:6789]" 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 3 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: ms bind msgr2 = False 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay max = 1 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay probability = 0.005 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay type = mon 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject internal delays = 0.002 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 2500 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-04-25T21:39:14.880 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-04-25T21:39:14.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-04-25T21:39:14.882 INFO:teuthology.orchestra.run.smithi203.stdout: bdev async discard = True 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: bdev enable discard = True 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 20 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 20 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-04-25T21:39:14.883 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: osd memory target = 1610612736 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-04-25T21:39:14.884 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 10 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-04-25T21:39:14.885 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-04-25T21:39:14.886 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 25 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject crc mismatch = 0.01 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject missing keys = 0.05 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 5 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T21:39:14.887 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-04-25T21:39:14.888 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-25T21:39:14.888 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-04-25T21:39:14.888 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-04-25T21:39:14.888 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-04-25T21:39:14.888 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T21:39:14.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T21:39:14.968 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T21:39:14.968 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:14.968 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T21:39:14.973 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:14.974 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T21:39:15.022 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:39:15.022 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:15.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T21:39:15.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T21:39:15.088 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:15.088 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:15.134 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:39:15.134 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:15.134 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T21:39:15.134 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T21:39:15.157 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:15.158 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:15.207 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T21:39:15.207 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T21:39:15.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T21:39:15.293 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T21:39:15.307 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T21:39:15.307 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T21:39:15.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T21:39:15.382 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T21:39:15.392 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T21:39:15.393 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-25T21:39:15.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T21:39:15.407 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:15.407 INFO:tasks.ceph:role: osd.0 2024-04-25T21:39:15.407 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:39:15.407 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T21:39:15.470 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T21:39:15.471 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T21:39:15.471 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T21:39:15.930 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-25T21:39:15.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-25T21:39:15.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T21:39:16.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T21:39:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:39:16.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T21:39:16.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T21:39:16.146 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:16.146 INFO:tasks.ceph:role: osd.1 2024-04-25T21:39:16.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:39:16.146 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T21:39:16.210 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T21:39:16.211 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T21:39:16.211 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T21:39:16.634 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-25T21:39:16.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-25T21:39:16.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T21:39:16.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T21:39:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:39:16.752 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T21:39:16.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T21:39:16.804 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:16.804 INFO:tasks.ceph:role: osd.2 2024-04-25T21:39:16.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:39:16.805 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T21:39:16.864 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T21:39:16.865 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T21:39:17.316 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-04-25T21:39:17.318 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-04-25T21:39:17.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T21:39:17.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T21:39:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:39:17.424 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T21:39:17.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:17.529+0000 7fd47e98e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T21:39:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:17.529+0000 7fd47e98e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T21:39:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:17.529+0000 7fd47e98e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T21:39:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:17.529+0000 7fd47e98e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T21:39:17.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:17.529+0000 7fd47e98e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T21:39:17.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:17.533+0000 7fd47e98e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T21:39:20.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T21:39:20.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:20.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:20.609+0000 7f79026087c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T21:39:20.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:20.613+0000 7f79026087c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T21:39:20.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:20.613+0000 7f79026087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T21:39:20.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:20.613+0000 7f79026087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T21:39:20.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:20.613+0000 7f79026087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T21:39:20.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:20.613+0000 7f79026087c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T21:39:23.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T21:39:23.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:23.717+0000 7f8dfcdb97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T21:39:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:23.721+0000 7f8dfcdb97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T21:39:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:23.721+0000 7f8dfcdb97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T21:39:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:23.721+0000 7f8dfcdb97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T21:39:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:23.721+0000 7f8dfcdb97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T21:39:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:23.721+0000 7f8dfcdb97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T21:39:26.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T21:39:26.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T21:39:26.793 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:26.793 INFO:tasks.ceph:role: osd.3 2024-04-25T21:39:26.793 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:39:26.793 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T21:39:26.853 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T21:39:26.854 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T21:39:27.338 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-25T21:39:27.340 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-25T21:39:27.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-25T21:39:27.496 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T21:39:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:39:27.509 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T21:39:27.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T21:39:27.561 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:27.561 INFO:tasks.ceph:role: osd.4 2024-04-25T21:39:27.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:39:27.561 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T21:39:27.626 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T21:39:27.626 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T21:39:27.626 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T21:39:27.626 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T21:39:27.626 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T21:39:27.626 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-25T21:39:27.627 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T21:39:27.627 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T21:39:27.627 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T21:39:27.627 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T21:39:28.071 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-25T21:39:28.073 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-25T21:39:28.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-25T21:39:28.167 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T21:39:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:39:28.177 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T21:39:28.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T21:39:28.227 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-25T21:39:28.227 INFO:tasks.ceph:role: osd.5 2024-04-25T21:39:28.227 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:39:28.228 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T21:39:28.291 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T21:39:28.755 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-25T21:39:28.757 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-25T21:39:28.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-25T21:39:28.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T21:39:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:39:28.860 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T21:39:28.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:28.972 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:28.968+0000 7ffb2a6b77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T21:39:28.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:28.972+0000 7ffb2a6b77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T21:39:28.975 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:28.972+0000 7ffb2a6b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T21:39:28.975 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:28.972+0000 7ffb2a6b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T21:39:28.975 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:28.972+0000 7ffb2a6b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T21:39:28.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:28.976+0000 7ffb2a6b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T21:39:31.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T21:39:31.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:32.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:32.052+0000 7f65a35877c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T21:39:32.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:32.052+0000 7f65a35877c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T21:39:32.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:32.052+0000 7f65a35877c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T21:39:32.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:32.052+0000 7f65a35877c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T21:39:32.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:32.052+0000 7f65a35877c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T21:39:32.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:32.052+0000 7f65a35877c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T21:39:35.019 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T21:39:35.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:35.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:35.156+0000 7f077ba8c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T21:39:35.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:35.156+0000 7f077ba8c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T21:39:35.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:35.156+0000 7f077ba8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T21:39:35.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:35.156+0000 7f077ba8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T21:39:35.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:35.156+0000 7f077ba8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T21:39:35.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:35.156+0000 7f077ba8c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T21:39:38.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T21:39:38.202 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T21:39:38.203 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:38.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T21:39:38.218 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:38.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T21:39:38.274 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:38.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T21:39:38.329 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:38.329 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T21:39:38.343 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:38.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T21:39:38.397 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:38.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T21:39:38.453 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:38.454 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T21:39:38.510 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:38.510 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T21:39:38.558 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T21:39:38.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T21:39:38.561 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T21:39:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-04-25T21:39:38.604 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCFzSpmojzBHxAAdT0VOCuix96mfEsfbjOGBQ== 2024-04-25T21:39:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-04-25T21:39:38.604 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCIzSpmTSSuJBAA9wK/7Vb9E40fiDVResn7BA== 2024-04-25T21:39:38.604 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCLzSpmsbUKKxAAqryhwV916kvS52y++UkyWQ== 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCDzSpmKOqBERAAJiKwdhLRaR8NKlxjZKsEOA== 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCQzSpmyeQaOhAA/hQpfCFJzeDKCL6Eak39+w== 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCUzSpmuy1xAxAAARNHvU/SgKZgt+Kt/WUDkQ== 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCXzSpmBu2vCRAAbtxCb2Iz8RFFGEOwtZcfpA== 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-04-25T21:39:38.605 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCDzSpm2rLuFhAAuALQmIPiW9C4AgKrPhMg8A== 2024-04-25T21:39:38.612 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-04-25T21:39:38.612 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCFzSpmojzBHxAAdT0VOCuix96mfEsfbjOGBQ== 2024-04-25T21:39:38.612 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-04-25T21:39:38.612 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCIzSpmTSSuJBAA9wK/7Vb9E40fiDVResn7BA== 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCLzSpmsbUKKxAAqryhwV916kvS52y++UkyWQ== 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCDzSpmKOqBERAAJiKwdhLRaR8NKlxjZKsEOA== 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCQzSpmyeQaOhAA/hQpfCFJzeDKCL6Eak39+w== 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCUzSpmuy1xAxAAARNHvU/SgKZgt+Kt/WUDkQ== 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCXzSpmBu2vCRAAbtxCb2Iz8RFFGEOwtZcfpA== 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-04-25T21:39:38.613 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCDzSpm2rLuFhAAuALQmIPiW9C4AgKrPhMg8A== 2024-04-25T21:39:38.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:38.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:38.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:38.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:38.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:38.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:38.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T21:39:38.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T21:39:38.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:38.965 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:39.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:39.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:39.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:39.135 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T21:39:39.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T21:39:39.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T21:39:39.268 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T21:39:39.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T21:39:39.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T21:39:39.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T21:39:39.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T21:39:39.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T21:39:39.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T21:39:39.584 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T21:39:39.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T21:39:39.691 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T21:39:39.702 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:39.704 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T21:39:39.750 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T21:39:39.750 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T21:39:39.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T21:39:39.753 INFO:tasks.ceph.mon.a:Started 2024-04-25T21:39:39.753 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T21:39:39.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T21:39:39.755 INFO:tasks.ceph.mon.c:Started 2024-04-25T21:39:39.755 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T21:39:39.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T21:39:39.795 INFO:tasks.ceph.mon.b:Started 2024-04-25T21:39:39.795 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T21:39:39.795 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T21:39:39.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T21:39:39.797 INFO:tasks.ceph.mgr.x:Started 2024-04-25T21:39:39.798 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T21:39:39.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T21:39:39.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient: get_monmap_and_config 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient: build_initial_monmap 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient: monmap: 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:39.919884+0000 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:39.919884+0000 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:39.918 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 1 Processor -- start 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 1 -- start start 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient: init 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient(hunting): picked mon.noname-a con 0x7f77540a3e80 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient(hunting): picked mon.noname-b con 0x7f77540a4970 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient(hunting): picked mon.noname-c con 0x7f775413d5a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775409adf0 con 0x7f77540a3e80 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775409b340 con 0x7f775413d5a0 2024-04-25T21:39:39.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775409b620 con 0x7f77540a4970 2024-04-25T21:39:39.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:39.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f775a878640 10 monclient(hunting): authenticate will time out at 938.829789s 2024-04-25T21:39:39.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f7753fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f77540a3e80 0x7f77540a4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50586/0 (socket says 172.21.15.6:50586) 2024-04-25T21:39:39.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.916+0000 7f7753fff640 1 -- 172.21.15.6:0/1723358918 learned_addr learned my addr 172.21.15.6:0/1723358918 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:39.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.920+0000 7f7753fff640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6789/0 conn(0x7f77540a3e80 legacy=0x7f77540a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:39.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.920+0000 7f774bfff640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6790/0 conn(0x7f77540a4970 legacy=0x7f775413be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:39.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:39.920+0000 7f7758e15640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.203:6789/0 conn(0x7f775413d5a0 legacy=0x7f775413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient(hunting): tick 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient(hunting): _check_auth_tickets 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 1 monclient(hunting): continuing hunt 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6790/0 conn(0x7f77540a4970 legacy=0x7f775413be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.203:6789/0 conn(0x7f775413d5a0 legacy=0x7f775413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6789/0 conn(0x7f77540a3e80 legacy=0x7f77540a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient(hunting): picked mon.noname-b con 0x7f7734003870 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:42.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient(hunting): picked mon.noname-c con 0x7f7734007530 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient(hunting): picked mon.noname-a con 0x7f773400b260 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775406eaf0 con 0x7f773400b260 2024-04-25T21:39:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773400ff90 con 0x7f7734007530 2024-04-25T21:39:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77340111c0 con 0x7f7734003870 2024-04-25T21:39:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f77527fc640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f7753fff640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6790/0 conn(0x7f7734003870 legacy=0x7f7734005cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f7758e15640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6789/0 conn(0x7f773400b260 legacy=0x7f773400d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:42.920+0000 7f774bfff640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.203:6789/0 conn(0x7f7734007530 legacy=0x7f77340099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient(hunting): tick 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient(hunting): _check_auth_tickets 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 1 monclient(hunting): continuing hunt 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6790/0 conn(0x7f7734003870 legacy=0x7f7734005cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.203:6789/0 conn(0x7f7734007530 legacy=0x7f77340099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6789/0 conn(0x7f773400b260 legacy=0x7f773400d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient(hunting): picked mon.noname-c con 0x7f773400f300 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient(hunting): picked mon.noname-a con 0x7f77340113c0 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient(hunting): picked mon.noname-b con 0x7f7734003870 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77340123f0 con 0x7f77340113c0 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77340120e0 con 0x7f773400f300 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773400b680 con 0x7f7734003870 2024-04-25T21:39:45.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.916+0000 7f77527fc640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:45.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.920+0000 7f774bfff640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6789/0 conn(0x7f77340113c0 legacy=0x7f7734011800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:45.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.920+0000 7f7753fff640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.203:6789/0 conn(0x7f773400f300 legacy=0x7f773400f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:45.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:45.920+0000 7f7758e15640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6790/0 conn(0x7f7734003870 legacy=0x7f7734005cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559549240 0 0) 0x7f77340123f0 con 0x7f77340113c0 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _negotiate 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): my global_id is 4098 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7740003530 con 0x7f77340113c0 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956252224 0 0) 0x7f77340120e0 con 0x7f773400f300 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _negotiate 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): my global_id is 4102 2024-04-25T21:39:46.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77340123f0 con 0x7f773400f300 2024-04-25T21:39:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1758315365 0 0) 0x7f773400b680 con 0x7f7734003870 2024-04-25T21:39:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _negotiate 2024-04-25T21:39:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient(hunting): my global_id is 4103 2024-04-25T21:39:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77340120e0 con 0x7f7734003870 2024-04-25T21:39:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2611710089 0 0) 0x7f77340123f0 con 0x7f773400f300 2024-04-25T21:39:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6790/0 conn(0x7f7734003870 legacy=0x7f7734005cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.6:6789/0 conn(0x7f77340113c0 legacy=0x7f7734011800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: _finish_hunting 0 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 monclient: found mon.noname-c 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773400c8a0 con 0x7f773400f300 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: _finish_auth 0 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: _check_auth_tickets 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f77440018a0 con 0x7f773400f300 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: dump: 2024-04-25T21:39:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f775a878640 5 monclient: authenticate success, global_id 4102 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f775a878640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7744001b40 con 0x7f773400f300 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 1 -- 172.21.15.6:0/1723358918 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f77440048b0 con 0x7f773400f300 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f77537fe640 10 monclient: dump: 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f775a878640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.992+0000 7f775a878640 4 set_mon_vals no callback set 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient: shutdown 2024-04-25T21:39:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- 172.21.15.6:0/1723358918 >> v1:172.21.15.203:6789/0 conn(0x7f773400f300 legacy=0x7f773400f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- 172.21.15.6:0/1723358918 shutdown_connections 2024-04-25T21:39:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- 172.21.15.6:0/1723358918 >> 172.21.15.6:0/1723358918 conn(0x7f77540a9e30 msgr2=0x7f77540a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- 172.21.15.6:0/1723358918 shutdown_connections 2024-04-25T21:39:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- 172.21.15.6:0/1723358918 wait complete. 2024-04-25T21:39:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient: build_initial_monmap 2024-04-25T21:39:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient: monmap: 2024-04-25T21:39:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:39:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 Processor -- start 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- start start 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient: init 2024-04-25T21:39:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient(hunting): picked mon.noname-c con 0x7f7734003870 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient(hunting): picked mon.noname-a con 0x7f773400f300 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient(hunting): picked mon.noname-b con 0x7f77340113c0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77541213e0 con 0x7f773400f300 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77541208f0 con 0x7f77340113c0 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7754121660 con 0x7f7734003870 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:46.996+0000 7f775a878640 10 monclient(hunting): authenticate will time out at 945.911983s 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f774bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f773400f300 0x7f77541201e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60728/0 (socket says 172.21.15.6:60728) 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f774bfff640 1 -- 172.21.15.6:0/3188610990 learned_addr learned my addr 172.21.15.6:0/3188610990 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f774bfff640 1 -- 172.21.15.6:0/3188610990 >> v1:172.21.15.6:6789/0 conn(0x7f773400f300 legacy=0x7f77541201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f7753fff640 1 -- 172.21.15.6:0/3188610990 >> v1:172.21.15.6:6790/0 conn(0x7f7734003870 legacy=0x7f775411fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1189172775 0 0) 0x7f77541213e0 con 0x7f773400f300 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): _negotiate 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): my global_id is 4104 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77340123f0 con 0x7f773400f300 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 452585315 0 0) 0x7f77340123f0 con 0x7f773400f300 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77541213e0 con 0x7f773400f300 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f773c001a00 con 0x7f773400f300 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): dump: 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f7758e15640 1 -- 172.21.15.6:0/3188610990 >> v1:172.21.15.203:6789/0 conn(0x7f77340113c0 legacy=0x7f77541d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1566892242 0 0) 0x7f77541213e0 con 0x7f773400f300 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 >> v1:172.21.15.6:6790/0 conn(0x7f7734003870 legacy=0x7f775411fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 >> v1:172.21.15.203:6789/0 conn(0x7f77340113c0 legacy=0x7f77541d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: _finish_hunting 0 2024-04-25T21:39:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 monclient: found mon.a 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7754122df0 con 0x7f773400f300 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: _finish_auth 0 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: _check_auth_tickets 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f775a878640 5 monclient: authenticate success, global_id 4104 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f775a878640 10 monclient: _renew_subs 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f775a878640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7754122550 con 0x7f773400f300 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f775a878640 10 monclient: _renew_subs 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f775a878640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f77541378d0 con 0x7f773400f300 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f773c0038d0 con 0x7f773400f300 2024-04-25T21:39:47.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f773c003c20 con 0x7f773400f300 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f7752ffd640 4 set_mon_vals no callback set 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 10 monclient: dump: 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f773c004190 con 0x7f773400f300 2024-04-25T21:39:47.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.000+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 2296295523 0 0) 0x7f773c006a00 con 0x7f773400f300 2024-04-25T21:39:47.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.004+0000 7f775a878640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:47.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.004+0000 7f775a878640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:47.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.004+0000 7f775a878640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:47.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.004+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7738005090 con 0x7f773400f300 2024-04-25T21:39:47.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.008+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f773c0057d0 con 0x7f773400f300 2024-04-25T21:39:47.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.008+0000 7f77517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:47.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.008+0000 7f77517fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.072+0000 7f775a878640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-25T21:39:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.072+0000 7f775a878640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-25T21:39:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.072+0000 7f775a878640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.072+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7738005380 con 0x7f773400f300 2024-04-25T21:39:47.198 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:47.196+0000 7f90eb2a2180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T21:39:47.686 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:47.684+0000 7f90eb2a2180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T21:39:47.874 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:47.872+0000 7f90eb2a2180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T21:39:47.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f77517fa640 1 -- 172.21.15.6:0/3188610990 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f773c005ab0 con 0x7f773400f300 2024-04-25T21:39:47.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f77517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-25T21:39:47.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f77517fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-25T21:39:47.946 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-04-25T21:39:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f775a878640 10 monclient: shutdown 2024-04-25T21:39:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f775a878640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 >> v1:172.21.15.6:6789/0 conn(0x7f773400f300 legacy=0x7f77541201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 shutdown_connections 2024-04-25T21:39:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 >> 172.21.15.6:0/3188610990 conn(0x7f77540a9e30 msgr2=0x7f77540a78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:47.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 shutdown_connections 2024-04-25T21:39:47.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:47.944+0000 7f775a878640 1 -- 172.21.15.6:0/3188610990 wait complete. 2024-04-25T21:39:47.971 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T21:39:47.971 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T21:39:47.972 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:47.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T21:39:47.985 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:47.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T21:39:48.042 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-04-25T21:39:48.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T21:39:48.097 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:48.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T21:39:48.110 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:48.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T21:39:48.165 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T21:39:48.165 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T21:39:48.220 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 2b3e085e-25b5-4eff-bed1-787528f6c0ba 0 2024-04-25T21:39:48.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient: get_monmap_and_config 2024-04-25T21:39:48.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient: build_initial_monmap 2024-04-25T21:39:48.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient: monmap: 2024-04-25T21:39:48.379 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:48.379 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:48.379 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:48.381479+0000 2024-04-25T21:39:48.380 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:48.381479+0000 2024-04-25T21:39:48.380 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:48.380 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:48.380 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:48.380 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:48.380 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:48.380 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:48.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 1 Processor -- start 2024-04-25T21:39:48.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 1 -- start start 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient: init 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient(hunting): picked mon.noname-c con 0x7fbf00136eb0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient(hunting): picked mon.noname-a con 0x7fbf00139c90 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient(hunting): picked mon.noname-b con 0x7fbf0013d7a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0009aa60 con 0x7fbf00139c90 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0009b820 con 0x7fbf00136eb0 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0006b2c0 con 0x7fbf0013d7a0 2024-04-25T21:39:48.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:48.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf07834640 10 monclient(hunting): authenticate will time out at 944.019523s 2024-04-25T21:39:48.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf055d0640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fbf00136eb0 0x7fbf00137290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59014/0 (socket says 172.21.15.203:59014) 2024-04-25T21:39:48.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.376+0000 7fbf055d0640 1 -- 172.21.15.203:0/723372411 learned_addr learned my addr 172.21.15.203:0/723372411 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:48.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf055d0640 1 -- 172.21.15.203:0/723372411 >> v1:172.21.15.203:6789/0 conn(0x7fbf00136eb0 legacy=0x7fbf00137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:48.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf04dcf640 1 -- 172.21.15.203:0/723372411 >> v1:172.21.15.6:6789/0 conn(0x7fbf00139c90 legacy=0x7fbf0013c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:48.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf05dd1640 1 -- 172.21.15.203:0/723372411 >> v1:172.21.15.6:6790/0 conn(0x7fbf0013d7a0 legacy=0x7fbf0013fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3008865203 0 0) 0x7fbf0009aa60 con 0x7fbf00139c90 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient(hunting): my global_id is 4107 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeec003530 con 0x7fbf00139c90 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1264266340 0 0) 0x7fbeec003530 con 0x7fbf00139c90 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 >> v1:172.21.15.6:6790/0 conn(0x7fbf0013d7a0 legacy=0x7fbf0013fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 >> v1:172.21.15.203:6789/0 conn(0x7fbf00136eb0 legacy=0x7fbf00137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: _finish_hunting 0 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 monclient: found mon.noname-a 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:39:48.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0009af80 con 0x7fbf00139c90 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: _finish_auth 0 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: _check_auth_tickets 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbef0001920 con 0x7fbf00139c90 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: dump: 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:48.387 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf07834640 5 monclient: authenticate success, global_id 4107 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf07834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbef0001c70 con 0x7fbf00139c90 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 1 -- 172.21.15.203:0/723372411 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbef00037c0 con 0x7fbf00139c90 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbef7fff640 10 monclient: dump: 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.388 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf07834640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf07834640 4 set_mon_vals no callback set 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf07834640 10 monclient: shutdown 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf07834640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.380+0000 7fbf07834640 1 -- 172.21.15.203:0/723372411 >> v1:172.21.15.6:6789/0 conn(0x7fbf00139c90 legacy=0x7fbf0013c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- 172.21.15.203:0/723372411 shutdown_connections 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- 172.21.15.203:0/723372411 >> 172.21.15.203:0/723372411 conn(0x7fbf000a69c0 msgr2=0x7fbf000a2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- 172.21.15.203:0/723372411 shutdown_connections 2024-04-25T21:39:48.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- 172.21.15.203:0/723372411 wait complete. 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient: build_initial_monmap 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient: monmap: 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-25T21:39:48.393 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 Processor -- start 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- start start 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient: init 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient(hunting): picked mon.noname-c con 0x7fbf00136eb0 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient(hunting): picked mon.noname-a con 0x7fbf00139c90 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient(hunting): picked mon.noname-b con 0x7fbf0013d7a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0013d350 con 0x7fbf00139c90 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf0011e080 con 0x7fbf0013d7a0 2024-04-25T21:39:48.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf00136c80 con 0x7fbf00136eb0 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf07834640 10 monclient(hunting): authenticate will time out at 944.025472s 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf05dd1640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fbf0013d7a0 0x7fbf001d43a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59018/0 (socket says 172.21.15.203:59018) 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf05dd1640 1 -- 172.21.15.203:0/3371353762 learned_addr learned my addr 172.21.15.203:0/3371353762 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.384+0000 7fbf05dd1640 1 -- 172.21.15.203:0/3371353762 >> v1:172.21.15.203:6789/0 conn(0x7fbf0013d7a0 legacy=0x7fbf001d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf04dcf640 1 -- 172.21.15.203:0/3371353762 >> v1:172.21.15.6:6789/0 conn(0x7fbf00139c90 legacy=0x7fbf001cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf055d0640 1 -- 172.21.15.203:0/3371353762 >> v1:172.21.15.6:6790/0 conn(0x7fbf00136eb0 legacy=0x7fbf0011c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 397801278 0 0) 0x7fbf0009af80 con 0x7fbf00139c90 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): my global_id is 4110 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbee4003530 con 0x7fbf00139c90 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2297976336 0 0) 0x7fbee4003530 con 0x7fbf00139c90 2024-04-25T21:39:48.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf0009af80 con 0x7fbf00139c90 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbef00037c0 con 0x7fbf00139c90 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): dump: 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:48.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1359666428 0 0) 0x7fbf0009af80 con 0x7fbf00139c90 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 >> v1:172.21.15.6:6790/0 conn(0x7fbf00136eb0 legacy=0x7fbf0011c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 >> v1:172.21.15.203:6789/0 conn(0x7fbf0013d7a0 legacy=0x7fbf001d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 monclient: found mon.a 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0011dac0 con 0x7fbf00139c90 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: _finish_auth 0 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 5 monclient: authenticate success, global_id 4110 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 10 monclient: _renew_subs 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0011dda0 con 0x7fbf00139c90 2024-04-25T21:39:48.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 10 monclient: _renew_subs 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf00135d30 con 0x7fbf00139c90 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbef0001c70 con 0x7fbf00139c90 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbef0003fd0 con 0x7fbf00139c90 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef6ffd640 4 set_mon_vals no callback set 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 10 monclient: dump: 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:48.398 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbef00057d0 con 0x7fbf00139c90 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 3917150197 0 0) 0x7fbef0006bd0 con 0x7fbf00139c90 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:48.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.388+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee0005090 con 0x7fbf00139c90 2024-04-25T21:39:48.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.404+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbf0013d350 con 0x7fbf00139c90 2024-04-25T21:39:48.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.404+0000 7fbef5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:48.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.404+0000 7fbef5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:48.425 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.424+0000 7f90eb2a2180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T21:39:48.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.472+0000 7fbf07834640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2b3e085e-25b5-4eff-bed1-787528f6c0ba", "id": 0}] 2024-04-25T21:39:48.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.472+0000 7fbf07834640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2b3e085e-25b5-4eff-bed1-787528f6c0ba", "id": 0}] 2024-04-25T21:39:48.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.472+0000 7fbf07834640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:48.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.472+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2b3e085e-25b5-4eff-bed1-787528f6c0ba", "id": 0} v 0) -- 0x7fbee0005380 con 0x7fbf00139c90 2024-04-25T21:39:48.482 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.476+0000 7f90eb2a2180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T21:39:48.532 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.528+0000 7f90eb2a2180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T21:39:48.607 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.604+0000 7f90eb2a2180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T21:39:48.659 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.656+0000 7f90eb2a2180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T21:39:48.735 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.732+0000 7f90eb2a2180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T21:39:48.787 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.784+0000 7f90eb2a2180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T21:39:48.934 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.932+0000 7f90eb2a2180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T21:39:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbef5ffb640 1 -- 172.21.15.203:0/3371353762 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b3e085e-25b5-4eff-bed1-787528f6c0ba", "id": 0}]=0 v3) ==== 112+0+2 (unknown 3951160066 0 732707515) 0x7fbef0005c40 con 0x7fbf00139c90 2024-04-25T21:39:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbef5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2b3e085e-25b5-4eff-bed1-787528f6c0ba", "id": 0}] 2024-04-25T21:39:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbef5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:48.935 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-04-25T21:39:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbf07834640 10 monclient: shutdown 2024-04-25T21:39:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbf07834640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 >> v1:172.21.15.6:6789/0 conn(0x7fbf00139c90 legacy=0x7fbf001cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 shutdown_connections 2024-04-25T21:39:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 >> 172.21.15.203:0/3371353762 conn(0x7fbf000a69c0 msgr2=0x7fbf00138070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:48.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 shutdown_connections 2024-04-25T21:39:48.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:48.932+0000 7fbf07834640 1 -- 172.21.15.203:0/3371353762 wait complete. 2024-04-25T21:39:48.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 6e5949dd-a414-4826-a94e-3e985f784fa7 1 2024-04-25T21:39:48.998 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:48.996+0000 7f90eb2a2180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T21:39:49.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient: get_monmap_and_config 2024-04-25T21:39:49.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient: monmap: 2024-04-25T21:39:49.067 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.067 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.067 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:49.069803+0000 2024-04-25T21:39:49.068 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:49.069803+0000 2024-04-25T21:39:49.068 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.068 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.068 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.068 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:49.068 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:49.068 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 1 Processor -- start 2024-04-25T21:39:49.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 1 -- start start 2024-04-25T21:39:49.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient: init 2024-04-25T21:39:49.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:39:49.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient(hunting): picked mon.noname-a con 0x7f4dac0a3030 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient(hunting): picked mon.noname-c con 0x7f4dac133960 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient(hunting): picked mon.noname-b con 0x7f4dac139ec0 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dac072810 con 0x7f4dac0a3030 2024-04-25T21:39:49.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dac070330 con 0x7f4dac133960 2024-04-25T21:39:49.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dac070610 con 0x7f4dac139ec0 2024-04-25T21:39:49.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4db1fee640 10 monclient(hunting): authenticate will time out at 944.707241s 2024-04-25T21:39:49.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4daaffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f4dac133960 0x7f4dac1387a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59022/0 (socket says 172.21.15.203:59022) 2024-04-25T21:39:49.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.064+0000 7f4daaffd640 1 -- 172.21.15.203:0/1450138507 learned_addr learned my addr 172.21.15.203:0/1450138507 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daaffd640 1 -- 172.21.15.203:0/1450138507 >> v1:172.21.15.203:6789/0 conn(0x7f4dac133960 legacy=0x7f4dac1387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1939327584 0 0) 0x7f4dac070330 con 0x7f4dac133960 2024-04-25T21:39:49.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient(hunting): my global_id is 4105 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d94003530 con 0x7f4dac133960 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4dab7fe640 1 -- 172.21.15.203:0/1450138507 >> v1:172.21.15.6:6789/0 conn(0x7f4dac0a3030 legacy=0x7f4dac072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4dabfff640 1 -- 172.21.15.203:0/1450138507 >> v1:172.21.15.6:6790/0 conn(0x7f4dac139ec0 legacy=0x7f4dac13c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2658134547 0 0) 0x7f4d94003530 con 0x7f4dac133960 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 >> v1:172.21.15.6:6790/0 conn(0x7f4dac139ec0 legacy=0x7f4dac13c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 >> v1:172.21.15.6:6789/0 conn(0x7f4dac0a3030 legacy=0x7f4dac072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 monclient: found mon.noname-c 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dac0708f0 con 0x7f4dac133960 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 5 monclient: authenticate success, global_id 4105 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4da00018d0 con 0x7f4dac133960 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: dump: 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.076 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4da0001c20 con 0x7f4dac133960 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 1 -- 172.21.15.203:0/1450138507 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4da0004a50 con 0x7f4dac133960 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 4 set_mon_vals no callback set 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4daa7fc640 10 monclient: dump: 2024-04-25T21:39:49.077 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 10 monclient: shutdown 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 1 -- 172.21.15.203:0/1450138507 >> v1:172.21.15.203:6789/0 conn(0x7f4dac133960 legacy=0x7f4dac1387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 1 -- 172.21.15.203:0/1450138507 shutdown_connections 2024-04-25T21:39:49.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 1 -- 172.21.15.203:0/1450138507 >> 172.21.15.203:0/1450138507 conn(0x7f4dac0a2540 msgr2=0x7f4dac0a2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 1 -- 172.21.15.203:0/1450138507 shutdown_connections 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 1 -- 172.21.15.203:0/1450138507 wait complete. 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 10 monclient: monmap: 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:49.079 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 1 Processor -- start 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 1 -- start start 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.068+0000 7f4db1fee640 10 monclient: init 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient(hunting): picked mon.noname-c con 0x7f4dac0a3030 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient(hunting): picked mon.noname-a con 0x7f4dac133960 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient(hunting): picked mon.noname-b con 0x7f4dac139ec0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dac09f4f0 con 0x7f4dac133960 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dac1ca800 con 0x7f4dac139ec0 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dac134000 con 0x7f4dac0a3030 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient(hunting): authenticate will time out at 944.712154s 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4dabfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f4dac139ec0 0x7f4dac1d2370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59036/0 (socket says 172.21.15.203:59036) 2024-04-25T21:39:49.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4dabfff640 1 -- 172.21.15.203:0/639840183 learned_addr learned my addr 172.21.15.203:0/639840183 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4dabfff640 1 -- 172.21.15.203:0/639840183 >> v1:172.21.15.203:6789/0 conn(0x7f4dac139ec0 legacy=0x7f4dac1d2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2290991370 0 0) 0x7f4dac1ca800 con 0x7f4dac139ec0 2024-04-25T21:39:49.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): my global_id is 4108 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d98003530 con 0x7f4dac139ec0 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4dab7fe640 1 -- 172.21.15.203:0/639840183 >> v1:172.21.15.6:6790/0 conn(0x7f4dac0a3030 legacy=0x7f4dac1ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4daaffd640 1 -- 172.21.15.203:0/639840183 >> v1:172.21.15.6:6789/0 conn(0x7f4dac133960 legacy=0x7f4dac1cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 110204074 0 0) 0x7f4d98003530 con 0x7f4dac139ec0 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4dac1ca800 con 0x7f4dac139ec0 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4d9c001e90 con 0x7f4dac139ec0 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): dump: 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.084 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:49.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3180928426 0 0) 0x7f4dac1ca800 con 0x7f4dac139ec0 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 >> v1:172.21.15.6:6790/0 conn(0x7f4dac0a3030 legacy=0x7f4dac1ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 >> v1:172.21.15.6:6789/0 conn(0x7f4dac133960 legacy=0x7f4dac1cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 monclient: found mon.b 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dac09f210 con 0x7f4dac139ec0 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 5 monclient: authenticate success, global_id 4108 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _renew_subs 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4dac09dd80 con 0x7f4dac139ec0 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _renew_subs 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4dac1c2890 con 0x7f4dac139ec0 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d9c003c50 con 0x7f4dac139ec0 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4d9c003fa0 con 0x7f4dac139ec0 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4da9ffb640 4 set_mon_vals no callback set 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 10 monclient: dump: 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.087 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4d9c0044d0 con 0x7f4dac139ec0 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 2850231908 0 0) 0x7f4d9c006920 con 0x7f4dac139ec0 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.072+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d8c005090 con 0x7f4dac139ec0 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.076+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4d9c005b50 con 0x7f4dac139ec0 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.076+0000 7f4d93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.076+0000 7f4d93fff640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:49.148 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.144+0000 7f4db1fee640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6e5949dd-a414-4826-a94e-3e985f784fa7", "id": 1}] 2024-04-25T21:39:49.148 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.144+0000 7f4db1fee640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6e5949dd-a414-4826-a94e-3e985f784fa7", "id": 1}] 2024-04-25T21:39:49.148 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.144+0000 7f4db1fee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.148 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.144+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6e5949dd-a414-4826-a94e-3e985f784fa7", "id": 1} v 0) -- 0x7f4d8c005380 con 0x7f4dac139ec0 2024-04-25T21:39:49.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4d93fff640 1 -- 172.21.15.203:0/639840183 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e5949dd-a414-4826-a94e-3e985f784fa7", "id": 1}]=0 v4) ==== 112+0+2 (unknown 3200726209 0 940483276) 0x7f4d9c005e80 con 0x7f4dac139ec0 2024-04-25T21:39:49.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4d93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6e5949dd-a414-4826-a94e-3e985f784fa7", "id": 1}] 2024-04-25T21:39:49.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4d93fff640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:49.158 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-04-25T21:39:49.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4db1fee640 10 monclient: shutdown 2024-04-25T21:39:49.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4db1fee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 >> v1:172.21.15.203:6789/0 conn(0x7f4dac139ec0 legacy=0x7f4dac1d2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 shutdown_connections 2024-04-25T21:39:49.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 >> 172.21.15.203:0/639840183 conn(0x7f4dac0a2540 msgr2=0x7f4dac06e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 shutdown_connections 2024-04-25T21:39:49.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.156+0000 7f4db1fee640 1 -- 172.21.15.203:0/639840183 wait complete. 2024-04-25T21:39:49.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 44114590-b8c2-4e3a-8073-aecac03aa2a1 2 2024-04-25T21:39:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.284+0000 7fb8b35b4640 10 monclient: get_monmap_and_config 2024-04-25T21:39:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.284+0000 7fb8b35b4640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.284+0000 7fb8b35b4640 10 monclient: monmap: 2024-04-25T21:39:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:49.291645+0000 2024-04-25T21:39:49.289 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:49.291645+0000 2024-04-25T21:39:49.290 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.290 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.290 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.290 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:49.290 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:49.290 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 1 Processor -- start 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 1 -- start start 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient: init 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient(hunting): picked mon.noname-c con 0x7fb8ac0a3e10 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ac0a5750 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient(hunting): picked mon.noname-a con 0x7fb8ac09bb90 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ac06eca0 con 0x7fb8ac09bb90 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ac09adf0 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ac09b0d0 con 0x7fb8ac0a5750 2024-04-25T21:39:49.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b35b4640 10 monclient(hunting): authenticate will time out at 944.929730s 2024-04-25T21:39:49.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b1350640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb8ac0a3e10 0x7fb8ac0a41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59038/0 (socket says 172.21.15.203:59038) 2024-04-25T21:39:49.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.288+0000 7fb8b1350640 1 -- 172.21.15.203:0/1276211445 learned_addr learned my addr 172.21.15.203:0/1276211445 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b1350640 1 -- 172.21.15.203:0/1276211445 >> v1:172.21.15.203:6789/0 conn(0x7fb8ac0a3e10 legacy=0x7fb8ac0a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3476411282 0 0) 0x7fb8ac09adf0 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): my global_id is 4111 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb898003530 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b0b4f640 1 -- 172.21.15.203:0/1276211445 >> v1:172.21.15.6:6790/0 conn(0x7fb8ac0a5750 legacy=0x7fb8ac13c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b1b51640 1 -- 172.21.15.203:0/1276211445 >> v1:172.21.15.6:6789/0 conn(0x7fb8ac09bb90 legacy=0x7fb8ac13f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675252665 0 0) 0x7fb8ac09b0d0 con 0x7fb8ac0a5750 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): my global_id is 4112 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ac09adf0 con 0x7fb8ac0a5750 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3813190258 0 0) 0x7fb8ac06eca0 con 0x7fb8ac09bb90 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient(hunting): my global_id is 4119 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ac09b0d0 con 0x7fb8ac09bb90 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1847645787 0 0) 0x7fb898003530 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 >> v1:172.21.15.6:6790/0 conn(0x7fb8ac0a5750 legacy=0x7fb8ac13c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 >> v1:172.21.15.6:6789/0 conn(0x7fb8ac09bb90 legacy=0x7fb8ac13f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 monclient: found mon.noname-c 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ac0a6880 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb8a4001c50 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 5 monclient: authenticate success, global_id 4111 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: dump: 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.297 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8a4001f20 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 4 set_mon_vals no callback set 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 1 -- 172.21.15.203:0/1276211445 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb8a4004dd0 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8a3fff640 10 monclient: dump: 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.298 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.299 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.299 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.299 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.299 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.299 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 10 monclient: shutdown 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 1 -- 172.21.15.203:0/1276211445 >> v1:172.21.15.203:6789/0 conn(0x7fb8ac0a3e10 legacy=0x7fb8ac0a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 1 -- 172.21.15.203:0/1276211445 shutdown_connections 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 1 -- 172.21.15.203:0/1276211445 >> 172.21.15.203:0/1276211445 conn(0x7fb8ac0a9cc0 msgr2=0x7fb8ac0a5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 1 -- 172.21.15.203:0/1276211445 shutdown_connections 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.292+0000 7fb8b35b4640 1 -- 172.21.15.203:0/1276211445 wait complete. 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient: monmap: 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.300 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 1 Processor -- start 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 1 -- start start 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient: init 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ac09bb90 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient(hunting): picked mon.noname-c con 0x7fb8ac0a3e10 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient(hunting): picked mon.noname-a con 0x7fb8ac0a5750 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ac1203d0 con 0x7fb8ac0a5750 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ac0a3320 con 0x7fb8ac09bb90 2024-04-25T21:39:49.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ac120be0 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b35b4640 10 monclient(hunting): authenticate will time out at 944.937833s 2024-04-25T21:39:49.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b1350640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb8ac09bb90 0x7fb8ac135120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59054/0 (socket says 172.21.15.203:59054) 2024-04-25T21:39:49.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.296+0000 7fb8b1350640 1 -- 172.21.15.203:0/3911422349 learned_addr learned my addr 172.21.15.203:0/3911422349 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b1350640 1 -- 172.21.15.203:0/3911422349 >> v1:172.21.15.203:6789/0 conn(0x7fb8ac09bb90 legacy=0x7fb8ac135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b0b4f640 1 -- 172.21.15.203:0/3911422349 >> v1:172.21.15.6:6790/0 conn(0x7fb8ac0a3e10 legacy=0x7fb8ac1cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2687800761 0 0) 0x7fb8ac0a3320 con 0x7fb8ac09bb90 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): my global_id is 4114 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb890003530 con 0x7fb8ac09bb90 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b1b51640 1 -- 172.21.15.203:0/3911422349 >> v1:172.21.15.6:6789/0 conn(0x7fb8ac0a5750 legacy=0x7fb8ac1cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249412402 0 0) 0x7fb8ac120be0 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): my global_id is 4115 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ac0a3320 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660050205 0 0) 0x7fb8ac1203d0 con 0x7fb8ac0a5750 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): my global_id is 4122 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8ac120be0 con 0x7fb8ac0a5750 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 9758997 0 0) 0x7fb890003530 con 0x7fb8ac09bb90 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8ac1203d0 con 0x7fb8ac09bb90 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb8a4004d10 con 0x7fb8ac09bb90 2024-04-25T21:39:49.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): dump: 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:49.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1720107515 0 0) 0x7fb8ac120be0 con 0x7fb8ac0a5750 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb890003530 con 0x7fb8ac0a5750 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 358344271 0 0) 0x7fb8ac0a3320 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb8ac120be0 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb8a8001a70 con 0x7fb8ac0a5750 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): dump: 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.307 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb89c001bc0 con 0x7fb8ac0a3e10 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient(hunting): dump: 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.308 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3596700113 0 0) 0x7fb8ac1203d0 con 0x7fb8ac09bb90 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 >> v1:172.21.15.6:6790/0 conn(0x7fb8ac0a3e10 legacy=0x7fb8ac1cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 >> v1:172.21.15.6:6789/0 conn(0x7fb8ac0a5750 legacy=0x7fb8ac1cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 monclient: found mon.b 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ac073530 con 0x7fb8ac09bb90 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b35b4640 5 monclient: authenticate success, global_id 4114 2024-04-25T21:39:49.309 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b35b4640 10 monclient: _renew_subs 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b35b4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ac120680 con 0x7fb8ac09bb90 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b35b4640 10 monclient: _renew_subs 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b35b4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb8ac1c4950 con 0x7fb8ac09bb90 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8a4001f20 con 0x7fb8ac09bb90 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.300+0000 7fb8a1ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb8a40055e0 con 0x7fb8ac09bb90 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8a1ffb640 10 monclient: dump: 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.310 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb8a4005b10 con 0x7fb8ac09bb90 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 1054746950 0 0) 0x7fb8a4006f40 con 0x7fb8ac09bb90 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8a37fe640 4 set_mon_vals no callback set 2024-04-25T21:39:49.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8b35b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8b35b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8b35b4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.304+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb88c005090 con 0x7fb8ac09bb90 2024-04-25T21:39:49.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.308+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb8a4006b20 con 0x7fb8ac09bb90 2024-04-25T21:39:49.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.308+0000 7fb8a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.308+0000 7fb8a1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:49.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.408+0000 7fb8b35b4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "44114590-b8c2-4e3a-8073-aecac03aa2a1", "id": 2}] 2024-04-25T21:39:49.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.408+0000 7fb8b35b4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "44114590-b8c2-4e3a-8073-aecac03aa2a1", "id": 2}] 2024-04-25T21:39:49.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.408+0000 7fb8b35b4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.408+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "44114590-b8c2-4e3a-8073-aecac03aa2a1", "id": 2} v 0) -- 0x7fb88c005380 con 0x7fb8ac09bb90 2024-04-25T21:39:49.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.416+0000 7fb8a1ffb640 1 -- 172.21.15.203:0/3911422349 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "44114590-b8c2-4e3a-8073-aecac03aa2a1", "id": 2}]=0 v5) ==== 112+0+2 (unknown 3435094460 0 216599125) 0x7fb8a4006d00 con 0x7fb8ac09bb90 2024-04-25T21:39:49.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.416+0000 7fb8a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "44114590-b8c2-4e3a-8073-aecac03aa2a1", "id": 2}] 2024-04-25T21:39:49.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.416+0000 7fb8a1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:49.422 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-04-25T21:39:49.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.420+0000 7fb8b35b4640 10 monclient: shutdown 2024-04-25T21:39:49.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.420+0000 7fb8b35b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.420+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 >> v1:172.21.15.203:6789/0 conn(0x7fb8ac09bb90 legacy=0x7fb8ac135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.420+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 shutdown_connections 2024-04-25T21:39:49.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.420+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 >> 172.21.15.203:0/3911422349 conn(0x7fb8ac0a9cc0 msgr2=0x7fb8ac136f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.420+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 shutdown_connections 2024-04-25T21:39:49.423 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.420+0000 7fb8b35b4640 1 -- 172.21.15.203:0/3911422349 wait complete. 2024-04-25T21:39:49.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 752f9b53-7f3b-4d16-8e45-84e24469a515 3 2024-04-25T21:39:49.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient: get_monmap_and_config 2024-04-25T21:39:49.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient: monmap: 2024-04-25T21:39:49.546 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.546 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.546 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:49.548694+0000 2024-04-25T21:39:49.547 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:49.548694+0000 2024-04-25T21:39:49.547 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.547 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.547 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.547 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:49.547 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:49.547 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 1 Processor -- start 2024-04-25T21:39:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 1 -- start start 2024-04-25T21:39:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient: init 2024-04-25T21:39:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:39:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient(hunting): picked mon.noname-c con 0x7f131c0a3e10 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient(hunting): picked mon.noname-b con 0x7f131c0a5750 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient(hunting): picked mon.noname-a con 0x7f131c09bb90 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c06eca0 con 0x7f131c09bb90 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c09adf0 con 0x7f131c0a3e10 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c09b0d0 con 0x7f131c0a5750 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f132180e640 10 monclient(hunting): authenticate will time out at 945.185922s 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f131affd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f131c0a3e10 0x7f131c0a41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59066/0 (socket says 172.21.15.203:59066) 2024-04-25T21:39:49.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.544+0000 7f131affd640 1 -- 172.21.15.203:0/4179362056 learned_addr learned my addr 172.21.15.203:0/4179362056 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f131affd640 1 -- 172.21.15.203:0/4179362056 >> v1:172.21.15.203:6789/0 conn(0x7f131c0a3e10 legacy=0x7f131c0a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3409922424 0 0) 0x7f131c09adf0 con 0x7f131c0a3e10 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient(hunting): my global_id is 4117 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1308003530 con 0x7f131c0a3e10 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f131a7fc640 1 -- 172.21.15.203:0/4179362056 >> v1:172.21.15.6:6790/0 conn(0x7f131c0a5750 legacy=0x7f131c13c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f131b7fe640 1 -- 172.21.15.203:0/4179362056 >> v1:172.21.15.6:6789/0 conn(0x7f131c09bb90 legacy=0x7f131c13f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4016009781 0 0) 0x7f1308003530 con 0x7f131c0a3e10 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 >> v1:172.21.15.6:6790/0 conn(0x7f131c0a5750 legacy=0x7f131c13c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 >> v1:172.21.15.6:6789/0 conn(0x7f131c09bb90 legacy=0x7f131c13f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 monclient: found mon.noname-c 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c0a6880 con 0x7f131c0a3e10 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 5 monclient: authenticate success, global_id 4117 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f13140018d0 con 0x7f131c0a3e10 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: dump: 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.553 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1314001c20 con 0x7f131c0a3e10 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 1 -- 172.21.15.203:0/4179362056 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f1314004a50 con 0x7f131c0a3e10 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 4 set_mon_vals no callback set 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f1319ffb640 10 monclient: dump: 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient: shutdown 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- 172.21.15.203:0/4179362056 >> v1:172.21.15.203:6789/0 conn(0x7f131c0a3e10 legacy=0x7f131c0a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- 172.21.15.203:0/4179362056 shutdown_connections 2024-04-25T21:39:49.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- 172.21.15.203:0/4179362056 >> 172.21.15.203:0/4179362056 conn(0x7f131c0a9cc0 msgr2=0x7f131c0a5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- 172.21.15.203:0/4179362056 shutdown_connections 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- 172.21.15.203:0/4179362056 wait complete. 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient: monmap: 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 Processor -- start 2024-04-25T21:39:49.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- start start 2024-04-25T21:39:49.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient: init 2024-04-25T21:39:49.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:39:49.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient(hunting): picked mon.noname-c con 0x7f131c09bb90 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient(hunting): picked mon.noname-a con 0x7f131c0a3e10 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient(hunting): picked mon.noname-b con 0x7f131c0a5750 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c13d520 con 0x7f131c0a3e10 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c13de00 con 0x7f131c0a5750 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131c13ef70 con 0x7f131c09bb90 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f132180e640 10 monclient(hunting): authenticate will time out at 945.190896s 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f131b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f131c0a5750 0x7f131c1d48c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59072/0 (socket says 172.21.15.203:59072) 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.548+0000 7f131b7fe640 1 -- 172.21.15.203:0/3131516010 learned_addr learned my addr 172.21.15.203:0/3131516010 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f131b7fe640 1 -- 172.21.15.203:0/3131516010 >> v1:172.21.15.203:6789/0 conn(0x7f131c0a5750 legacy=0x7f131c1d48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047636463 0 0) 0x7f131c13de00 con 0x7f131c0a5750 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): my global_id is 4120 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12fc003530 con 0x7f131c0a5750 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2219798806 0 0) 0x7f12fc003530 con 0x7f131c0a5750 2024-04-25T21:39:49.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f131c13de00 con 0x7f131c0a5750 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f130c001e90 con 0x7f131c0a5750 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): dump: 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.562 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f131affd640 1 -- 172.21.15.203:0/3131516010 >> v1:172.21.15.6:6790/0 conn(0x7f131c09bb90 legacy=0x7f131c1cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f131a7fc640 1 -- 172.21.15.203:0/3131516010 >> v1:172.21.15.6:6789/0 conn(0x7f131c0a3e10 legacy=0x7f131c1d1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 359099839 0 0) 0x7f131c13de00 con 0x7f131c0a5750 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 >> v1:172.21.15.6:6790/0 conn(0x7f131c09bb90 legacy=0x7f131c1cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 >> v1:172.21.15.6:6789/0 conn(0x7f131c0a3e10 legacy=0x7f131c1d1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 monclient: found mon.b 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c137d10 con 0x7f131c0a5750 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 5 monclient: authenticate success, global_id 4120 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 10 monclient: _renew_subs 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f131c13c890 con 0x7f131c0a5750 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 10 monclient: _renew_subs 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f131c13fc30 con 0x7f131c0a5750 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f130c003c50 con 0x7f131c0a5750 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f130c003fa0 con 0x7f131c0a5750 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f13197fa640 4 set_mon_vals no callback set 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 10 monclient: dump: 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f130c004590 con 0x7f131c0a5750 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 1738233390 0 0) 0x7f130c0070d0 con 0x7f131c0a5750 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.552+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1300005090 con 0x7f131c0a5750 2024-04-25T21:39:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.556+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f130c006d50 con 0x7f131c0a5750 2024-04-25T21:39:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.556+0000 7f1313fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.556+0000 7f1313fff640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:49.610 INFO:tasks.ceph.mgr.x.smithi203.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-25T21:39:49.610 INFO:tasks.ceph.mgr.x.smithi203.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-25T21:39:49.610 INFO:tasks.ceph.mgr.x.smithi203.stderr: from numpy import show_config as show_numpy_config 2024-04-25T21:39:49.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.620+0000 7f132180e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "752f9b53-7f3b-4d16-8e45-84e24469a515", "id": 3}] 2024-04-25T21:39:49.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.620+0000 7f132180e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "752f9b53-7f3b-4d16-8e45-84e24469a515", "id": 3}] 2024-04-25T21:39:49.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.620+0000 7f132180e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.620+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "752f9b53-7f3b-4d16-8e45-84e24469a515", "id": 3} v 0) -- 0x7f1300005380 con 0x7f131c0a5750 2024-04-25T21:39:49.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f1313fff640 1 -- 172.21.15.203:0/3131516010 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "752f9b53-7f3b-4d16-8e45-84e24469a515", "id": 3}]=0 v6) ==== 112+0+2 (unknown 784227995 0 525046306) 0x7f130c005c90 con 0x7f131c0a5750 2024-04-25T21:39:49.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f1313fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "752f9b53-7f3b-4d16-8e45-84e24469a515", "id": 3}] 2024-04-25T21:39:49.635 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f1313fff640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:49.636 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-04-25T21:39:49.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f132180e640 10 monclient: shutdown 2024-04-25T21:39:49.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f132180e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 >> v1:172.21.15.203:6789/0 conn(0x7f131c0a5750 legacy=0x7f131c1d48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 shutdown_connections 2024-04-25T21:39:49.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 >> 172.21.15.203:0/3131516010 conn(0x7f131c0a9cc0 msgr2=0x7f131c0a2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 shutdown_connections 2024-04-25T21:39:49.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.632+0000 7f132180e640 1 -- 172.21.15.203:0/3131516010 wait complete. 2024-04-25T21:39:49.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 63bc5f2b-57c1-432c-bd64-3c3753ceb48f 4 2024-04-25T21:39:49.656 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:49.652+0000 7f90eb2a2180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T21:39:49.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient: get_monmap_and_config 2024-04-25T21:39:49.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient: monmap: 2024-04-25T21:39:49.763 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.763 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.763 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:49.765656+0000 2024-04-25T21:39:49.763 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:49.765656+0000 2024-04-25T21:39:49.764 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.764 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.764 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.764 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:49.764 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:49.764 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 1 Processor -- start 2024-04-25T21:39:49.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 1 -- start start 2024-04-25T21:39:49.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient: init 2024-04-25T21:39:49.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient(hunting): picked mon.noname-b con 0x7f380c136eb0 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient(hunting): picked mon.noname-c con 0x7f380c139c90 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient(hunting): picked mon.noname-a con 0x7f380c13d7a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380c09aa60 con 0x7f380c13d7a0 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380c09b820 con 0x7f380c139c90 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380c06b2c0 con 0x7f380c136eb0 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f3812f17640 10 monclient(hunting): authenticate will time out at 945.403059s 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f380bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f380c139c90 0x7f380c13c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59086/0 (socket says 172.21.15.203:59086) 2024-04-25T21:39:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.760+0000 7f380bfff640 1 -- 172.21.15.203:0/883527950 learned_addr learned my addr 172.21.15.203:0/883527950 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380bfff640 1 -- 172.21.15.203:0/883527950 >> v1:172.21.15.203:6789/0 conn(0x7f380c139c90 legacy=0x7f380c13c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1481289990 0 0) 0x7f380c09b820 con 0x7f380c139c90 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient(hunting): my global_id is 4123 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37f4003530 con 0x7f380c139c90 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3810cb3640 1 -- 172.21.15.203:0/883527950 >> v1:172.21.15.6:6790/0 conn(0x7f380c136eb0 legacy=0x7f380c137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f38114b4640 1 -- 172.21.15.203:0/883527950 >> v1:172.21.15.6:6789/0 conn(0x7f380c13d7a0 legacy=0x7f380c13fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1429003175 0 0) 0x7f37f4003530 con 0x7f380c139c90 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 >> v1:172.21.15.6:6790/0 conn(0x7f380c136eb0 legacy=0x7f380c137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 >> v1:172.21.15.6:6789/0 conn(0x7f380c13d7a0 legacy=0x7f380c13fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 monclient: found mon.noname-c 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380c09af80 con 0x7f380c139c90 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3800001990 con 0x7f380c139c90 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 5 monclient: authenticate success, global_id 4123 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: dump: 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3800001bf0 con 0x7f380c139c90 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 1 -- 172.21.15.203:0/883527950 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3800004a50 con 0x7f380c139c90 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 4 set_mon_vals no callback set 2024-04-25T21:39:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f380b7fe640 10 monclient: dump: 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 10 monclient: shutdown 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 1 -- 172.21.15.203:0/883527950 >> v1:172.21.15.203:6789/0 conn(0x7f380c139c90 legacy=0x7f380c13c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 1 -- 172.21.15.203:0/883527950 shutdown_connections 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 1 -- 172.21.15.203:0/883527950 >> 172.21.15.203:0/883527950 conn(0x7f380c0a69c0 msgr2=0x7f380c0a2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 1 -- 172.21.15.203:0/883527950 shutdown_connections 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 1 -- 172.21.15.203:0/883527950 wait complete. 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 10 monclient: build_initial_monmap 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 10 monclient: monmap: 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-25T21:39:49.773 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 1 Processor -- start 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 1 -- start start 2024-04-25T21:39:49.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.764+0000 7f3812f17640 10 monclient: init 2024-04-25T21:39:49.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:49.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:39:49.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient(hunting): picked mon.noname-b con 0x7f380c136eb0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:49.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient(hunting): picked mon.noname-c con 0x7f380c139c90 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:49.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient(hunting): picked mon.noname-a con 0x7f380c13d7a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:49.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380c1ccf60 con 0x7f380c13d7a0 2024-04-25T21:39:49.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380c136c80 con 0x7f380c136eb0 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f380c13d350 con 0x7f380c139c90 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient(hunting): authenticate will time out at 945.408199s 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3810cb3640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f380c136eb0 0x7f380c1cc140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59098/0 (socket says 172.21.15.203:59098) 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3810cb3640 1 -- 172.21.15.203:0/3901669773 learned_addr learned my addr 172.21.15.203:0/3901669773 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3810cb3640 1 -- 172.21.15.203:0/3901669773 >> v1:172.21.15.203:6789/0 conn(0x7f380c136eb0 legacy=0x7f380c1cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713473347 0 0) 0x7f380c136c80 con 0x7f380c136eb0 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): _negotiate 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): my global_id is 4126 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37f8003530 con 0x7f380c136eb0 2024-04-25T21:39:49.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3430572197 0 0) 0x7f37f8003530 con 0x7f380c136eb0 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f380c136c80 con 0x7f380c136eb0 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f37fc001ba0 con 0x7f380c136eb0 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): dump: 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.779 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38114b4640 1 -- 172.21.15.203:0/3901669773 >> v1:172.21.15.6:6789/0 conn(0x7f380c13d7a0 legacy=0x7f380c1d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f380bfff640 1 -- 172.21.15.203:0/3901669773 >> v1:172.21.15.6:6790/0 conn(0x7f380c139c90 legacy=0x7f380c1cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3510592894 0 0) 0x7f380c136c80 con 0x7f380c136eb0 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 >> v1:172.21.15.6:6790/0 conn(0x7f380c139c90 legacy=0x7f380c1cc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 >> v1:172.21.15.6:6789/0 conn(0x7f380c13d7a0 legacy=0x7f380c1d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: _finish_hunting 0 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 monclient: found mon.b 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380c13a8f0 con 0x7f380c136eb0 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: _finish_auth 0 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: _check_auth_tickets 2024-04-25T21:39:49.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 5 monclient: authenticate success, global_id 4126 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _renew_subs 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f380c0a02a0 con 0x7f380c136eb0 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _renew_subs 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f380c13ff00 con 0x7f380c136eb0 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37fc0038d0 con 0x7f380c136eb0 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f37fc003c20 con 0x7f380c136eb0 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 10 monclient: dump: 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:49.781 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f380affd640 4 set_mon_vals no callback set 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f37fc004150 con 0x7f380c136eb0 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 526740374 0 0) 0x7f37fc006670 con 0x7f380c136eb0 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.768+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ec005090 con 0x7f380c136eb0 2024-04-25T21:39:49.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.772+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f37fc006960 con 0x7f380c136eb0 2024-04-25T21:39:49.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.772+0000 7f38097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:49.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.772+0000 7f38097fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:49.793 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:49.788+0000 7f90eb2a2180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T21:39:49.843 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.840+0000 7f3812f17640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "63bc5f2b-57c1-432c-bd64-3c3753ceb48f", "id": 4}] 2024-04-25T21:39:49.843 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.840+0000 7f3812f17640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "63bc5f2b-57c1-432c-bd64-3c3753ceb48f", "id": 4}] 2024-04-25T21:39:49.843 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.840+0000 7f3812f17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-04-25T21:39:49.843 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.840+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "63bc5f2b-57c1-432c-bd64-3c3753ceb48f", "id": 4} v 0) -- 0x7f37ec005380 con 0x7f380c136eb0 2024-04-25T21:39:49.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.856+0000 7f38097fa640 1 -- 172.21.15.203:0/3901669773 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "63bc5f2b-57c1-432c-bd64-3c3753ceb48f", "id": 4}]=0 v7) ==== 112+0+2 (unknown 3322527968 0 1697012583) 0x7f37fc006c10 con 0x7f380c136eb0 2024-04-25T21:39:49.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.856+0000 7f38097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "63bc5f2b-57c1-432c-bd64-3c3753ceb48f", "id": 4}] 2024-04-25T21:39:49.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.856+0000 7f38097fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:49.861 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-04-25T21:39:49.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.856+0000 7f3812f17640 10 monclient: shutdown 2024-04-25T21:39:49.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.856+0000 7f3812f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:49.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.856+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 >> v1:172.21.15.203:6789/0 conn(0x7f380c136eb0 legacy=0x7f380c1cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:49.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.860+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 shutdown_connections 2024-04-25T21:39:49.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.860+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 >> 172.21.15.203:0/3901669773 conn(0x7f380c0a69c0 msgr2=0x7f380c130040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:49.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.860+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 shutdown_connections 2024-04-25T21:39:49.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:49.860+0000 7f3812f17640 1 -- 172.21.15.203:0/3901669773 wait complete. 2024-04-25T21:39:49.877 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 77440232-26ad-4654-9cba-3444b17e5b1b 5 2024-04-25T21:39:49.958 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:49.956+0000 7f90eb2a2180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T21:39:50.031 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient: get_monmap_and_config 2024-04-25T21:39:50.031 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient: build_initial_monmap 2024-04-25T21:39:50.031 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient: monmap: 2024-04-25T21:39:50.031 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:50.031 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.031 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:50.033711+0000 2024-04-25T21:39:50.032 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:50.033711+0000 2024-04-25T21:39:50.032 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:50.032 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:50.032 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:50.032 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:50.032 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:50.032 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:50.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 1 Processor -- start 2024-04-25T21:39:50.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 1 -- start start 2024-04-25T21:39:50.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient: init 2024-04-25T21:39:50.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient(hunting): picked mon.noname-a con 0x7f63c80a3e80 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient(hunting): picked mon.noname-b con 0x7f63c80a4970 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient(hunting): picked mon.noname-c con 0x7f63c813d5a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c809adf0 con 0x7f63c80a3e80 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c809b340 con 0x7f63c813d5a0 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c809b620 con 0x7f63c80a4970 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63cffdb640 10 monclient(hunting): authenticate will time out at 945.670980s 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63ce578640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f63c813d5a0 0x7f63c813f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59110/0 (socket says 172.21.15.203:59110) 2024-04-25T21:39:50.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.028+0000 7f63ce578640 1 -- 172.21.15.203:0/591083744 learned_addr learned my addr 172.21.15.203:0/591083744 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:50.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ce578640 1 -- 172.21.15.203:0/591083744 >> v1:172.21.15.203:6789/0 conn(0x7f63c813d5a0 legacy=0x7f63c813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cd576640 1 -- 172.21.15.203:0/591083744 >> v1:172.21.15.6:6790/0 conn(0x7f63c80a4970 legacy=0x7f63c813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cdd77640 1 -- 172.21.15.203:0/591083744 >> v1:172.21.15.6:6789/0 conn(0x7f63c80a3e80 legacy=0x7f63c80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992764873 0 0) 0x7f63c809b620 con 0x7f63c80a4970 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): my global_id is 4130 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63b4003530 con 0x7f63c80a4970 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3451533805 0 0) 0x7f63c809adf0 con 0x7f63c80a3e80 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient(hunting): my global_id is 4137 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63c809b620 con 0x7f63c80a3e80 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 223716085 0 0) 0x7f63b4003530 con 0x7f63c80a4970 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 >> v1:172.21.15.203:6789/0 conn(0x7f63c813d5a0 legacy=0x7f63c813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 >> v1:172.21.15.6:6789/0 conn(0x7f63c80a3e80 legacy=0x7f63c80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: _finish_hunting 0 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 monclient: found mon.noname-b 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c806eaf0 con 0x7f63c80a4970 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: _finish_auth 0 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: _check_auth_tickets 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f63c40019e0 con 0x7f63c80a4970 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:50.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: dump: 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 5 monclient: authenticate success, global_id 4130 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f63c4001d40 con 0x7f63c80a4970 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 4 set_mon_vals no callback set 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 1 -- 172.21.15.203:0/591083744 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f63c4003840 con 0x7f63c80a4970 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63ccd75640 10 monclient: dump: 2024-04-25T21:39:50.041 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 10 monclient: shutdown 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 1 -- 172.21.15.203:0/591083744 >> v1:172.21.15.6:6790/0 conn(0x7f63c80a4970 legacy=0x7f63c813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 1 -- 172.21.15.203:0/591083744 shutdown_connections 2024-04-25T21:39:50.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 1 -- 172.21.15.203:0/591083744 >> 172.21.15.203:0/591083744 conn(0x7f63c80a9e30 msgr2=0x7f63c80a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 1 -- 172.21.15.203:0/591083744 shutdown_connections 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 1 -- 172.21.15.203:0/591083744 wait complete. 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 10 monclient: build_initial_monmap 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.032+0000 7f63cffdb640 10 monclient: monmap: 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-04-25T21:39:50.043 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:50.044 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:50.044 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:50.044 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:50.044 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f90eb2a2180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T21:39:50.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 1 Processor -- start 2024-04-25T21:39:50.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 1 -- start start 2024-04-25T21:39:50.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient: init 2024-04-25T21:39:50.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:50.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient(hunting): picked mon.noname-c con 0x7f63c80a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient(hunting): picked mon.noname-a con 0x7f63c80a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient(hunting): picked mon.noname-b con 0x7f63c813d5a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c813e7a0 con 0x7f63c80a4970 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c809fba0 con 0x7f63c813d5a0 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c813fff0 con 0x7f63c80a3e80 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cffdb640 10 monclient(hunting): authenticate will time out at 945.676832s 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63ce578640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f63c813d5a0 0x7f63c81d4320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59118/0 (socket says 172.21.15.203:59118) 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63ce578640 1 -- 172.21.15.203:0/1957174796 learned_addr learned my addr 172.21.15.203:0/1957174796 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63ce578640 1 -- 172.21.15.203:0/1957174796 >> v1:172.21.15.203:6789/0 conn(0x7f63c813d5a0 legacy=0x7f63c81d4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cd576640 1 -- 172.21.15.203:0/1957174796 >> v1:172.21.15.6:6789/0 conn(0x7f63c80a4970 legacy=0x7f63c813e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63cdd77640 1 -- 172.21.15.203:0/1957174796 >> v1:172.21.15.6:6790/0 conn(0x7f63c80a3e80 legacy=0x7f63c813d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352365884 0 0) 0x7f63c813e7a0 con 0x7f63c80a4970 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): my global_id is 4140 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63ac003530 con 0x7f63c80a4970 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2797283326 0 0) 0x7f63c813fff0 con 0x7f63c80a3e80 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 10 monclient(hunting): my global_id is 4133 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.036+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63c813e7a0 con 0x7f63c80a3e80 2024-04-25T21:39:50.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3023859370 0 0) 0x7f63ac003530 con 0x7f63c80a4970 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63c813fff0 con 0x7f63c80a4970 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f63c40019a0 con 0x7f63c80a4970 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient(hunting): dump: 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.051 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2312718178 0 0) 0x7f63c813e7a0 con 0x7f63c80a3e80 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f63ac003530 con 0x7f63c80a3e80 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f63b8001ca0 con 0x7f63c80a3e80 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient(hunting): dump: 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.052 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2656172884 0 0) 0x7f63c813fff0 con 0x7f63c80a4970 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 >> v1:172.21.15.6:6790/0 conn(0x7f63c80a3e80 legacy=0x7f63c813d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 >> v1:172.21.15.203:6789/0 conn(0x7f63c813d5a0 legacy=0x7f63c81d4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: _finish_hunting 0 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 monclient: found mon.a 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c81402d0 con 0x7f63c80a4970 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: _finish_auth 0 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: _check_auth_tickets 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:50.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 5 monclient: authenticate success, global_id 4140 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 10 monclient: _renew_subs 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f63c81405b0 con 0x7f63c80a4970 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 10 monclient: _renew_subs 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f63c8137ba0 con 0x7f63c80a4970 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f63c4001d40 con 0x7f63c80a4970 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f63c4003fa0 con 0x7f63c80a4970 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 10 monclient: dump: 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-04-25T21:39:50.054 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63bffff640 4 set_mon_vals no callback set 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f63c40057d0 con 0x7f63c80a4970 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 839213661 0 0) 0x7f63c4006040 con 0x7f63c80a4970 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:50.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.040+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63b0005090 con 0x7f63c80a4970 2024-04-25T21:39:50.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.044+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f63c4006360 con 0x7f63c80a4970 2024-04-25T21:39:50.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.044+0000 7f63be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:50.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.044+0000 7f63be7fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:50.113 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.108+0000 7f63cffdb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "77440232-26ad-4654-9cba-3444b17e5b1b", "id": 5}] 2024-04-25T21:39:50.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.108+0000 7f63cffdb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "77440232-26ad-4654-9cba-3444b17e5b1b", "id": 5}] 2024-04-25T21:39:50.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.108+0000 7f63cffdb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:50.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.112+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd new", "uuid": "77440232-26ad-4654-9cba-3444b17e5b1b", "id": 5} v 0) -- 0x7f63b0005380 con 0x7f63c80a4970 2024-04-25T21:39:50.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63be7fc640 1 -- 172.21.15.203:0/1957174796 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "77440232-26ad-4654-9cba-3444b17e5b1b", "id": 5}]=0 v8) ==== 112+0+2 (unknown 1706852025 0 1988412176) 0x7f63c4005ac0 con 0x7f63c80a4970 2024-04-25T21:39:50.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "77440232-26ad-4654-9cba-3444b17e5b1b", "id": 5}] 2024-04-25T21:39:50.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63be7fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:50.122 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-04-25T21:39:50.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63cffdb640 10 monclient: shutdown 2024-04-25T21:39:50.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63cffdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:50.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 >> v1:172.21.15.6:6789/0 conn(0x7f63c80a4970 legacy=0x7f63c813e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 shutdown_connections 2024-04-25T21:39:50.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 >> 172.21.15.203:0/1957174796 conn(0x7f63c80a9e30 msgr2=0x7f63c80a7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:50.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 shutdown_connections 2024-04-25T21:39:50.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T21:39:50.120+0000 7f63cffdb640 1 -- 172.21.15.203:0/1957174796 wait complete. 2024-04-25T21:39:50.138 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T21:39:50.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T21:39:50.141 INFO:tasks.ceph.osd.0:Started 2024-04-25T21:39:50.141 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T21:39:50.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T21:39:50.143 INFO:tasks.ceph.osd.1:Started 2024-04-25T21:39:50.143 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T21:39:50.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T21:39:50.145 INFO:tasks.ceph.osd.2:Started 2024-04-25T21:39:50.146 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T21:39:50.146 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T21:39:50.177 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:50.172+0000 7f90eb2a2180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T21:39:50.179 INFO:tasks.ceph.osd.3:Started 2024-04-25T21:39:50.179 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T21:39:50.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T21:39:50.181 INFO:tasks.ceph.osd.4:Started 2024-04-25T21:39:50.181 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T21:39:50.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T21:39:50.184 INFO:tasks.ceph.osd.5:Started 2024-04-25T21:39:50.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T21:39:50.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.296+0000 7f5cbfc0a640 10 monclient: get_monmap_and_config 2024-04-25T21:39:50.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.296+0000 7f5cbfc0a640 10 monclient: build_initial_monmap 2024-04-25T21:39:50.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.296+0000 7f5cbfc0a640 10 monclient: monmap: 2024-04-25T21:39:50.300 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:39:50.300 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.300 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:50.302355+0000 2024-04-25T21:39:50.301 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:50.302355+0000 2024-04-25T21:39:50.301 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:50.301 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:50.301 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:50.301 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:50.301 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:50.301 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.296+0000 7f5cbfc0a640 1 Processor -- start 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 1 -- start start 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient: init 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient(hunting): picked mon.noname-c con 0x7f5cb8074b10 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient(hunting): picked mon.noname-a con 0x7f5cb806e9e0 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:50.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient(hunting): picked mon.noname-b con 0x7f5cb8139f70 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb8072a00 con 0x7f5cb806e9e0 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb806edf0 con 0x7f5cb8074b10 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb806f500 con 0x7f5cb8139f70 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient(hunting): authenticate will time out at 949.212722s 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbd1a5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5cb806e9e0 0x7f5cb8138850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60780/0 (socket says 172.21.15.6:60780) 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbe1a7640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5cb8139f70 0x7f5cb813c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49210/0 (socket says 172.21.15.6:49210) 2024-04-25T21:39:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbd1a5640 1 -- 172.21.15.6:0/908458145 learned_addr learned my addr 172.21.15.6:0/908458145 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:50.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbd1a5640 1 -- 172.21.15.6:0/908458145 >> v1:172.21.15.6:6789/0 conn(0x7f5cb806e9e0 legacy=0x7f5cb8138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbe1a7640 1 -- 172.21.15.6:0/908458145 >> v1:172.21.15.6:6790/0 conn(0x7f5cb8139f70 legacy=0x7f5cb813c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbd9a6640 1 -- 172.21.15.6:0/908458145 >> v1:172.21.15.203:6789/0 conn(0x7f5cb8074b10 legacy=0x7f5cb80722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 114334637 0 0) 0x7f5cb8072a00 con 0x7f5cb806e9e0 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): my global_id is 4161 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ca4003530 con 0x7f5cb806e9e0 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202684102 0 0) 0x7f5cb806f500 con 0x7f5cb8139f70 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient(hunting): my global_id is 4154 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cb8072a00 con 0x7f5cb8139f70 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3579021734 0 0) 0x7f5ca4003530 con 0x7f5cb806e9e0 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 >> v1:172.21.15.6:6790/0 conn(0x7f5cb8139f70 legacy=0x7f5cb813c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 >> v1:172.21.15.203:6789/0 conn(0x7f5cb8074b10 legacy=0x7f5cb80722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: _finish_hunting 0 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 monclient: found mon.noname-a 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb80a2110 con 0x7f5cb806e9e0 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: _finish_auth 0 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: _check_auth_tickets 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5ca80018d0 con 0x7f5cb806e9e0 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 5 monclient: authenticate success, global_id 4161 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: dump: 2024-04-25T21:39:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbd1a5640 1 --1- 172.21.15.6:0/908458145 >> v1:172.21.15.6:6789/0 conn(0x7f5cb806e9e0 0x7f5cb8138850 :-1 s=OPENED pgs=41 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f5ca8001c20 config(0 keys) 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ca8001c20 con 0x7f5cb806e9e0 2024-04-25T21:39:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbc9a4640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.300+0000 7f5cbfc0a640 4 set_mon_vals no callback set 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient: shutdown 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/908458145 >> v1:172.21.15.6:6789/0 conn(0x7f5cb806e9e0 legacy=0x7f5cb8138850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbc9a4640 1 -- 172.21.15.6:0/908458145 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5ca8004a50 con 0x7f5cb806e9e0 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbc9a4640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/908458145 shutdown_connections 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/908458145 >> 172.21.15.6:0/908458145 conn(0x7f5cb80a6000 msgr2=0x7f5cb80a6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/908458145 shutdown_connections 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/908458145 wait complete. 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient: build_initial_monmap 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient: monmap: 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 Processor -- start 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- start start 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient: init 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient(hunting): picked mon.noname-b con 0x7f5cb806e9e0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient(hunting): picked mon.noname-c con 0x7f5cb8074b10 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient(hunting): picked mon.noname-a con 0x7f5cb8139f70 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb80a00a0 con 0x7f5cb8139f70 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb81239d0 con 0x7f5cb806e9e0 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb8125490 con 0x7f5cb8074b10 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbfc0a640 10 monclient(hunting): authenticate will time out at 949.218121s 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbd1a5640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5cb8074b10 0x7f5cb81ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49212/0 (socket says 172.21.15.6:49212) 2024-04-25T21:39:50.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.304+0000 7f5cbd1a5640 1 -- 172.21.15.6:0/1808646316 learned_addr learned my addr 172.21.15.6:0/1808646316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbe1a7640 1 -- 172.21.15.6:0/1808646316 >> v1:172.21.15.6:6789/0 conn(0x7f5cb8139f70 legacy=0x7f5cb81d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbd1a5640 1 -- 172.21.15.6:0/1808646316 >> v1:172.21.15.6:6790/0 conn(0x7f5cb8074b10 legacy=0x7f5cb81ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbd9a6640 1 -- 172.21.15.6:0/1808646316 >> v1:172.21.15.203:6789/0 conn(0x7f5cb806e9e0 legacy=0x7f5cb8139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950756007 0 0) 0x7f5cb80a00a0 con 0x7f5cb8139f70 2024-04-25T21:39:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): my global_id is 4164 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c9c003560 con 0x7f5cb8139f70 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 59061554 0 0) 0x7f5cb8125490 con 0x7f5cb8074b10 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): my global_id is 4157 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cb80a00a0 con 0x7f5cb8074b10 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2751989864 0 0) 0x7f5cb80a00a0 con 0x7f5cb8074b10 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cb8125490 con 0x7f5cb8074b10 2024-04-25T21:39:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5ca8004990 con 0x7f5cb8074b10 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): dump: 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2040682226 0 0) 0x7f5c9c003560 con 0x7f5cb8139f70 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cb80a00a0 con 0x7f5cb8139f70 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5cb0001f40 con 0x7f5cb8139f70 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient(hunting): dump: 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3668219953 0 0) 0x7f5cb8125490 con 0x7f5cb8074b10 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 >> v1:172.21.15.203:6789/0 conn(0x7f5cb806e9e0 legacy=0x7f5cb8139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 >> v1:172.21.15.6:6789/0 conn(0x7f5cb8139f70 legacy=0x7f5cb81d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: _finish_hunting 0 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 monclient: found mon.c 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb8125d30 con 0x7f5cb8074b10 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: _finish_auth 0 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: _check_auth_tickets 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 5 monclient: authenticate success, global_id 4157 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 10 monclient: _renew_subs 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb8124750 con 0x7f5cb8074b10 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 10 monclient: _renew_subs 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5cb8139c40 con 0x7f5cb8074b10 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ca8001c20 con 0x7f5cb8074b10 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5ca8005270 con 0x7f5cb8074b10 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5caffff640 4 set_mon_vals no callback set 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 10 monclient: dump: 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5ca80057a0 con 0x7f5cb8074b10 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 2015940757 0 0) 0x7f5ca8006080 con 0x7f5cb8074b10 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.308+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ca0005090 con 0x7f5cb8074b10 2024-04-25T21:39:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.312+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5ca8005a90 con 0x7f5cb8074b10 2024-04-25T21:39:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.312+0000 7f5cae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.312+0000 7f5cae7fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:50.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:50.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:50.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:50.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ca0005380 con 0x7f5cb8074b10 2024-04-25T21:39:50.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cae7fc640 1 -- 172.21.15.6:0/1808646316 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 3141951485) 0x7f5ca8005d30 con 0x7f5cb8074b10 2024-04-25T21:39:50.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:50.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cae7fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:50.372 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:39:50.372 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:39:50.116466+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:39:50.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 10 monclient: shutdown 2024-04-25T21:39:50.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:50.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 >> v1:172.21.15.6:6790/0 conn(0x7f5cb8074b10 legacy=0x7f5cb81ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:50.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 shutdown_connections 2024-04-25T21:39:50.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 >> 172.21.15.6:0/1808646316 conn(0x7f5cb80a6000 msgr2=0x7f5cb809f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:50.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 shutdown_connections 2024-04-25T21:39:50.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:39:50.368+0000 7f5cbfc0a640 1 -- 172.21.15.6:0/1808646316 wait complete. 2024-04-25T21:39:50.389 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T21:39:50.389 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T21:39:50.514 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-25T21:39:50.512+0000 7f2dd7a547c0 -1 Falling back to public interface 2024-04-25T21:39:50.763 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-25T21:39:50.760+0000 7fe82a2167c0 -1 Falling back to public interface 2024-04-25T21:39:50.763 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-25T21:39:50.760+0000 7f4c7877e7c0 -1 Falling back to public interface 2024-04-25T21:39:50.791 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-04-25T21:39:50.788+0000 7fd3dea8d7c0 -1 Falling back to public interface 2024-04-25T21:39:50.791 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-25T21:39:50.788+0000 7fa5b883b7c0 -1 Falling back to public interface 2024-04-25T21:39:50.802 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-25T21:39:50.800+0000 7efd6bb4b7c0 -1 Falling back to public interface 2024-04-25T21:39:50.805 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:50.800+0000 7f90eb2a2180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T21:39:50.880 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:50.876+0000 7f90eb2a2180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T21:39:50.994 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T21:39:51.060 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:51.056+0000 7f90eb2a2180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T21:39:51.115 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient: get_monmap_and_config 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient: build_initial_monmap 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient: monmap: 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:51.117767+0000 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:51.117767+0000 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:51.116 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 1 Processor -- start 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 1 -- start start 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient: init 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient(hunting): picked mon.noname-b con 0x7fdedc0a3df0 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient(hunting): picked mon.noname-c con 0x7fdedc0a41d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient(hunting): picked mon.noname-a con 0x7fdedc13d010 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc0a02a0 con 0x7fdedc13d010 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc0a3850 con 0x7fdedc0a41d0 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc0a0a20 con 0x7fdedc0a3df0 2024-04-25T21:39:51.118 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:51.119 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee3612640 10 monclient(hunting): authenticate will time out at 950.027582s 2024-04-25T21:39:51.119 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee13ae640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdedc0a3df0 0x7fdedc0a1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:49214/0 (socket says 172.21.15.6:49214) 2024-04-25T21:39:51.119 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee13ae640 1 -- 172.21.15.6:0/1660477680 learned_addr learned my addr 172.21.15.6:0/1660477680 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:51.119 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.112+0000 7fdee1baf640 1 --1- 172.21.15.6:0/1660477680 >> v1:172.21.15.6:6789/0 conn(0x7fdedc13d010 0x7fdedc13f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60796/0 (socket says 172.21.15.6:60796) 2024-04-25T21:39:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee1baf640 1 -- 172.21.15.6:0/1660477680 >> v1:172.21.15.6:6789/0 conn(0x7fdedc13d010 legacy=0x7fdedc13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee13ae640 1 -- 172.21.15.6:0/1660477680 >> v1:172.21.15.6:6790/0 conn(0x7fdedc0a3df0 legacy=0x7fdedc0a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee0bad640 1 -- 172.21.15.6:0/1660477680 >> v1:172.21.15.203:6789/0 conn(0x7fdedc0a41d0 legacy=0x7fdedc0a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020509229 0 0) 0x7fdedc0a0a20 con 0x7fdedc0a3df0 2024-04-25T21:39:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:51.123 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): my global_id is 4160 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdec8003530 con 0x7fdedc0a3df0 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3114097421 0 0) 0x7fdedc0a02a0 con 0x7fdedc13d010 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): my global_id is 4167 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdedc0a0a20 con 0x7fdedc13d010 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198408039 0 0) 0x7fdedc0a3850 con 0x7fdedc0a41d0 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:51.124 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient(hunting): my global_id is 4129 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdedc0a02a0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 727834830 0 0) 0x7fdec8003530 con 0x7fdedc0a3df0 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 >> v1:172.21.15.203:6789/0 conn(0x7fdedc0a41d0 legacy=0x7fdedc0a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 >> v1:172.21.15.6:6789/0 conn(0x7fdedc13d010 legacy=0x7fdedc13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: _finish_hunting 0 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 monclient: found mon.noname-b 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc0a0d00 con 0x7fdedc0a3df0 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: _finish_auth 0 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: _check_auth_tickets 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fded80018d0 con 0x7fdedc0a3df0 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 5 monclient: authenticate success, global_id 4160 2024-04-25T21:39:51.125 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: dump: 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.126 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fded8001c20 con 0x7fdedc0a3df0 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 1 -- 172.21.15.6:0/1660477680 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fded8004a50 con 0x7fdedc0a3df0 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 4 set_mon_vals no callback set 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fded3fff640 10 monclient: dump: 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:51.127 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 10 monclient: shutdown 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 1 -- 172.21.15.6:0/1660477680 >> v1:172.21.15.6:6790/0 conn(0x7fdedc0a3df0 legacy=0x7fdedc0a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 1 -- 172.21.15.6:0/1660477680 shutdown_connections 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 1 -- 172.21.15.6:0/1660477680 >> 172.21.15.6:0/1660477680 conn(0x7fdedc0a5ef0 msgr2=0x7fdedc0a62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 1 -- 172.21.15.6:0/1660477680 shutdown_connections 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.116+0000 7fdee3612640 1 -- 172.21.15.6:0/1660477680 wait complete. 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient: build_initial_monmap 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient: monmap: 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:51.128 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 1 Processor -- start 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 1 -- start start 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient: init 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient(hunting): picked mon.noname-c con 0x7fdedc0a3df0 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient(hunting): picked mon.noname-a con 0x7fdedc0a41d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient(hunting): picked mon.noname-b con 0x7fdedc13d010 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:51.129 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc131020 con 0x7fdedc0a41d0 2024-04-25T21:39:51.130 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc131da0 con 0x7fdedc13d010 2024-04-25T21:39:51.130 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc1337d0 con 0x7fdedc0a3df0 2024-04-25T21:39:51.130 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:51.130 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee3612640 10 monclient(hunting): authenticate will time out at 950.032891s 2024-04-25T21:39:51.130 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee0bad640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdedc0a41d0 0x7fdedc130910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60808/0 (socket says 172.21.15.6:60808) 2024-04-25T21:39:51.130 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee0bad640 1 -- 172.21.15.6:0/4043390967 learned_addr learned my addr 172.21.15.6:0/4043390967 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee0bad640 1 -- 172.21.15.6:0/4043390967 >> v1:172.21.15.6:6789/0 conn(0x7fdedc0a41d0 legacy=0x7fdedc130910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee13ae640 1 -- 172.21.15.6:0/4043390967 >> v1:172.21.15.6:6790/0 conn(0x7fdedc0a3df0 legacy=0x7fdedc130200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1070620014 0 0) 0x7fdedc131020 con 0x7fdedc0a41d0 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fdee1baf640 1 -- 172.21.15.6:0/4043390967 >> v1:172.21.15.203:6789/0 conn(0x7fdedc13d010 legacy=0x7fdedc1d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): my global_id is 4170 2024-04-25T21:39:51.134 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdec0003530 con 0x7fdedc0a41d0 2024-04-25T21:39:51.135 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316622978 0 0) 0x7fdedc1337d0 con 0x7fdedc0a3df0 2024-04-25T21:39:51.135 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:51.135 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:51.135 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:51.135 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): my global_id is 4163 2024-04-25T21:39:51.135 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdedc131020 con 0x7fdedc0a3df0 2024-04-25T21:39:51.135 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1043794986 0 0) 0x7fdedc131da0 con 0x7fdedc13d010 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): my global_id is 4132 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdedc1337d0 con 0x7fdedc13d010 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3407931776 0 0) 0x7fdec0003530 con 0x7fdedc0a41d0 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdedc131da0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdecc001c50 con 0x7fdedc0a41d0 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): dump: 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:51.138 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1706991421 0 0) 0x7fdedc131020 con 0x7fdedc0a3df0 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdec0003530 con 0x7fdedc0a3df0 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fded8004990 con 0x7fdedc0a3df0 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:51.139 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): dump: 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3042548113 0 0) 0x7fdedc1337d0 con 0x7fdedc13d010 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdedc131020 con 0x7fdedc13d010 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fded4001fc0 con 0x7fdedc13d010 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:51.140 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 10 monclient(hunting): dump: 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2048021193 0 0) 0x7fdedc131da0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 >> v1:172.21.15.6:6790/0 conn(0x7fdedc0a3df0 legacy=0x7fdedc130200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.120+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 >> v1:172.21.15.203:6789/0 conn(0x7fdedc13d010 legacy=0x7fdedc1d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:39:51.141 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 1 monclient: found mon.a 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc132620 con 0x7fdedc0a41d0 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: _finish_auth 0 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 5 monclient: authenticate success, global_id 4170 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 10 monclient: _renew_subs 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc1358d0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 10 monclient: _renew_subs 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdedc1349c0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.142 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdecc003a90 con 0x7fdedc0a41d0 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded37fe640 4 set_mon_vals no callback set 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdecc003de0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: dump: 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.143 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdecc004310 con 0x7fdedc0a41d0 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 2015940757 0 0) 0x7fdecc005dc0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdebc005090 con 0x7fdedc0a41d0 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdecc006060 con 0x7fdedc0a41d0 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:51.144 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.124+0000 7fded1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:51.145 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:51.132+0000 7f90eb2a2180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T21:39:51.186 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:51.186 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:51.186 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:39:51.186 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdebc005380 con 0x7fdedc0a41d0 2024-04-25T21:39:51.187 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fded1ffb640 1 -- 172.21.15.6:0/4043390967 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 3141951485) 0x7fdecc0062c0 con 0x7fdedc0a41d0 2024-04-25T21:39:51.187 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fded1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:51.187 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fded1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:51.187 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-25T21:39:51.187 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":8,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:39:50.116466+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:39:51.188 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 10 monclient: shutdown 2024-04-25T21:39:51.188 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:51.188 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 >> v1:172.21.15.6:6789/0 conn(0x7fdedc0a41d0 legacy=0x7fdedc130910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:51.188 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 shutdown_connections 2024-04-25T21:39:51.188 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 >> 172.21.15.6:0/4043390967 conn(0x7fdedc0a5ef0 msgr2=0x7fdedc0a7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:51.188 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 shutdown_connections 2024-04-25T21:39:51.189 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:51.184+0000 7fdee3612640 1 -- 172.21.15.6:0/4043390967 wait complete. 2024-04-25T21:39:51.203 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-25T21:39:51.420 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:51.416+0000 7f90eb2a2180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T21:39:51.481 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:51.480+0000 7f90eb2a2180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T21:39:52.531 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:52.528+0000 7f90926e9640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T21:39:52.682 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-25T21:39:52.528+0000 7f90926e9640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T21:39:54.348 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-25T21:39:54.344+0000 7f2dd7a547c0 -1 osd.2 0 log_to_monitors true 2024-04-25T21:39:54.520 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-25T21:39:54.516+0000 7fe82a2167c0 -1 osd.0 0 log_to_monitors true 2024-04-25T21:39:54.592 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-25T21:39:54.588+0000 7efd6bb4b7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T21:39:54.609 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-25T21:39:54.604+0000 7f4c7877e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T21:39:54.692 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-04-25T21:39:54.688+0000 7fd3dea8d7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T21:39:54.699 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-25T21:39:54.696+0000 7fa5b883b7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T21:39:55.950 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-04-25T21:39:55.948+0000 7fd3da213640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T21:39:55.964 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-04-25T21:39:55.960+0000 7fd3d6862640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T21:39:57.290 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-25T21:39:57.288+0000 7fa5b57f7640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T21:39:57.294 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-04-25T21:39:57.292+0000 7fa5ac608640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T21:39:57.308 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-25T21:39:57.304+0000 7efd672d1640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T21:39:57.313 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-04-25T21:39:57.312+0000 7efd5f918640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T21:39:57.705 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-25T21:39:57.700+0000 7f2dd31da640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T21:39:57.711 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-04-25T21:39:57.708+0000 7f2dcb821640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T21:39:57.721 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-25T21:39:57.720+0000 7f4c73f04640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T21:39:57.727 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-25T21:39:57.724+0000 7f4c73f04640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T21:39:57.728 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-25T21:39:57.724+0000 7f4c73f04640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T21:39:57.728 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-25T21:39:57.724+0000 7f4c7573a640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T21:39:57.729 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-04-25T21:39:57.724+0000 7f4c6c54b640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T21:39:57.756 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-25T21:39:57.752+0000 7fe8271d2640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T21:39:57.761 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-04-25T21:39:57.756+0000 7fe81dfe3640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T21:39:57.808 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T21:39:57.928 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.924+0000 7fa453d8e640 10 monclient: get_monmap_and_config 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.924+0000 7fa453d8e640 10 monclient: build_initial_monmap 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.924+0000 7fa453d8e640 10 monclient: monmap: 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:57.930718+0000 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:57.930718+0000 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:39:57.929 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:57.931 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 1 Processor -- start 2024-04-25T21:39:57.931 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 1 -- start start 2024-04-25T21:39:57.931 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient: init 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient(hunting): picked mon.noname-b con 0x7fa44c074b80 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient(hunting): picked mon.noname-c con 0x7fa44c072700 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient(hunting): picked mon.noname-a con 0x7fa44c133ac0 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c0a4620 con 0x7fa44c133ac0 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c0a41e0 con 0x7fa44c072700 2024-04-25T21:39:57.932 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c06e4e0 con 0x7fa44c074b80 2024-04-25T21:39:57.933 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:57.933 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa453d8e640 10 monclient(hunting): authenticate will time out at 956.841329s 2024-04-25T21:39:57.933 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa45232b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa44c133ac0 0x7fa44c13f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43560/0 (socket says 172.21.15.6:43560) 2024-04-25T21:39:57.933 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa45232b640 1 -- 172.21.15.6:0/2725387727 learned_addr learned my addr 172.21.15.6:0/2725387727 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:57.936 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa45232b640 1 -- 172.21.15.6:0/2725387727 >> v1:172.21.15.6:6789/0 conn(0x7fa44c133ac0 legacy=0x7fa44c13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:57.936 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa451b2a640 1 -- 172.21.15.6:0/2725387727 >> v1:172.21.15.6:6790/0 conn(0x7fa44c074b80 legacy=0x7fa44c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917027702 0 0) 0x7fa44c0a4620 con 0x7fa44c133ac0 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): _negotiate 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): my global_id is 4206 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa438003530 con 0x7fa44c133ac0 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa451329640 1 -- 172.21.15.6:0/2725387727 >> v1:172.21.15.203:6789/0 conn(0x7fa44c072700 legacy=0x7fa44c13be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4021632388 0 0) 0x7fa44c06e4e0 con 0x7fa44c074b80 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): _negotiate 2024-04-25T21:39:57.937 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 10 monclient(hunting): my global_id is 4190 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.928+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa44c0a4620 con 0x7fa44c074b80 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2646881667 0 0) 0x7fa438003530 con 0x7fa44c133ac0 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 >> v1:172.21.15.6:6790/0 conn(0x7fa44c074b80 legacy=0x7fa44c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 >> v1:172.21.15.203:6789/0 conn(0x7fa44c072700 legacy=0x7fa44c13be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: _finish_hunting 0 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 monclient: found mon.noname-a 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44c0a4c70 con 0x7fa44c133ac0 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: _finish_auth 0 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: _check_auth_tickets 2024-04-25T21:39:57.938 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa444001c50 con 0x7fa44c133ac0 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: dump: 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:57.939 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 5 monclient: authenticate success, global_id 4206 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa444001fa0 con 0x7fa44c133ac0 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 1 -- 172.21.15.6:0/2725387727 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa444004dd0 con 0x7fa44c133ac0 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient: get_monmap_and_config success 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 4 set_mon_vals no callback set 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa450b28640 10 monclient: dump: 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:57.940 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient: shutdown 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- 172.21.15.6:0/2725387727 >> v1:172.21.15.6:6789/0 conn(0x7fa44c133ac0 legacy=0x7fa44c13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- 172.21.15.6:0/2725387727 shutdown_connections 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- 172.21.15.6:0/2725387727 >> 172.21.15.6:0/2725387727 conn(0x7fa44c0a5dd0 msgr2=0x7fa44c0a61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- 172.21.15.6:0/2725387727 shutdown_connections 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- 172.21.15.6:0/2725387727 wait complete. 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient: build_initial_monmap 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient: monmap: 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-04-25T21:39:57.941 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 Processor -- start 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- start start 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient: init 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient: _reopen_session rank -1 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient(hunting): picked mon.noname-b con 0x7fa44c072700 addr v1:172.21.15.203:6789/0 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient(hunting): picked mon.noname-c con 0x7fa44c074b80 addr v1:172.21.15.6:6790/0 2024-04-25T21:39:57.942 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient(hunting): picked mon.noname-a con 0x7fa44c133ac0 addr v1:172.21.15.6:6789/0 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c12e710 con 0x7fa44c133ac0 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c0a0e90 con 0x7fa44c072700 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44c130a70 con 0x7fa44c074b80 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient(hunting): _renew_subs 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa453d8e640 10 monclient(hunting): authenticate will time out at 956.846460s 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa45232b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa44c133ac0 0x7fa44c1d3f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43574/0 (socket says 172.21.15.6:43574) 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa45232b640 1 -- 172.21.15.6:0/546675713 learned_addr learned my addr 172.21.15.6:0/546675713 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:39:57.943 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.932+0000 7fa451329640 1 --1- 172.21.15.6:0/546675713 >> v1:172.21.15.6:6790/0 conn(0x7fa44c074b80 0x7fa44c12d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:60504/0 (socket says 172.21.15.6:60504) 2024-04-25T21:39:57.946 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa45232b640 1 -- 172.21.15.6:0/546675713 >> v1:172.21.15.6:6789/0 conn(0x7fa44c133ac0 legacy=0x7fa44c1d3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:57.946 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa451329640 1 -- 172.21.15.6:0/546675713 >> v1:172.21.15.6:6790/0 conn(0x7fa44c074b80 legacy=0x7fa44c12d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:57.946 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998340170 0 0) 0x7fa44c12e710 con 0x7fa44c133ac0 2024-04-25T21:39:57.946 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:57.946 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:57.946 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): my global_id is 4209 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa430003530 con 0x7fa44c133ac0 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482433966 0 0) 0x7fa44c130a70 con 0x7fa44c074b80 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): my global_id is 4193 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa44c12e710 con 0x7fa44c074b80 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa451b2a640 1 -- 172.21.15.6:0/546675713 >> v1:172.21.15.203:6789/0 conn(0x7fa44c072700 legacy=0x7fa44c12cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1554939037 0 0) 0x7fa430003530 con 0x7fa44c133ac0 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa44c130a70 con 0x7fa44c133ac0 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2055968490 0 0) 0x7fa44c12e710 con 0x7fa44c074b80 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa430003530 con 0x7fa44c074b80 2024-04-25T21:39:57.947 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa444004d10 con 0x7fa44c133ac0 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): dump: 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa43c001ad0 con 0x7fa44c074b80 2024-04-25T21:39:57.948 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): dump: 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2194325850 0 0) 0x7fa44c0a4620 con 0x7fa44c072700 2024-04-25T21:39:57.949 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _negotiate 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient(hunting): my global_id is 4171 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa44c12e710 con 0x7fa44c072700 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1470086629 0 0) 0x7fa430003530 con 0x7fa44c074b80 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 >> v1:172.21.15.203:6789/0 conn(0x7fa44c072700 legacy=0x7fa44c12cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 >> v1:172.21.15.6:6789/0 conn(0x7fa44c133ac0 legacy=0x7fa44c1d3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: _finish_hunting 0 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 monclient: found mon.c 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44c1301d0 con 0x7fa44c074b80 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: _finish_auth 0 2024-04-25T21:39:57.950 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: _check_auth_tickets 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 5 monclient: authenticate success, global_id 4193 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 10 monclient: _renew_subs 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa44c127e20 con 0x7fa44c074b80 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 10 monclient: _renew_subs 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa44c12dce0 con 0x7fa44c074b80 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa43c0038d0 con 0x7fa44c074b80 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa43c003c20 con 0x7fa44c074b80 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:39:57.951 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa443fff640 4 set_mon_vals no callback set 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 10 monclient: dump: 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 4) ==== 43962+0+0 (unknown 3775152483 0 0) 0x7fa43c003dc0 con 0x7fa44c074b80 2024-04-25T21:39:57.952 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa434005090 con 0x7fa44c074b80 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.936+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(11..11 src has 1..11) ==== 2067+0+0 (unknown 876530218 0 0) 0x7fa43c043b10 con 0x7fa44c074b80 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.940+0000 7fa451b2a640 10 monclient: get_auth_request con 0x7fa430036c00 auth_method 0 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.940+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fa43c0154b0 con 0x7fa44c074b80 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.940+0000 7fa4427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:39:57.953 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:57.940+0000 7fa4427fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:39:58.109 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.104+0000 7fa453d8e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:58.109 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.104+0000 7fa453d8e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:58.109 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.104+0000 7fa453d8e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:39:58.109 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.104+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa434005380 con 0x7fa44c074b80 2024-04-25T21:39:58.110 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa4427fc640 1 -- 172.21.15.6:0/546675713 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+5871 (unknown 238102980 0 1789341089) 0x7fa43c019160 con 0x7fa44c074b80 2024-04-25T21:39:58.110 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa4427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:39:58.110 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa4427fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:39:58.110 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-25T21:39:58.110 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:39:56.923603+0000","last_up_change":"2024-04-25T21:39:56.923603+0000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3201437079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3201437079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":3201437079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":3201437079}]},"public_addr":"172.21.15.203:6800/3201437079","cluster_addr":"172.21.15.203:6802/3201437079","heartbeat_back_addr":"172.21.15.203:6806/3201437079","heartbeat_front_addr":"172.21.15.203:6804/3201437079","state":["exists","up"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:39:58.112 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 >> v1:172.21.15.203:6812/3164820986 conn(0x7fa430036c00 legacy=0x7fa4300390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:58.112 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 10 monclient: shutdown 2024-04-25T21:39:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:39:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 >> v1:172.21.15.6:6790/0 conn(0x7fa44c074b80 legacy=0x7fa44c12d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:39:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 shutdown_connections 2024-04-25T21:39:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 >> 172.21.15.6:0/546675713 conn(0x7fa44c0a5dd0 msgr2=0x7fa44c132380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:39:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 shutdown_connections 2024-04-25T21:39:58.113 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:39:58.108+0000 7fa453d8e640 1 -- 172.21.15.6:0/546675713 wait complete. 2024-04-25T21:39:58.127 DEBUG:teuthology.misc:1 of 6 OSDs are up 2024-04-25T21:40:04.733 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.852+0000 7ffa2214c640 10 monclient: get_monmap_and_config 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.852+0000 7ffa2214c640 10 monclient: build_initial_monmap 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.852+0000 7ffa2214c640 10 monclient: monmap: 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:40:04.859180+0000 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:40:04.859180+0000 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:04.857 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:04.858 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:04.858 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:04.858 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:04.858 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 1 Processor -- start 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 1 -- start start 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient: init 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1c0a2770 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1c0a1270 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:04.859 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1c0a16b0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:04.860 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c09a690 con 0x7ffa1c0a16b0 2024-04-25T21:40:04.860 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c09eef0 con 0x7ffa1c0a1270 2024-04-25T21:40:04.860 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c099820 con 0x7ffa1c0a2770 2024-04-25T21:40:04.860 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:04.860 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa2214c640 10 monclient(hunting): authenticate will time out at 963.769168s 2024-04-25T21:40:04.860 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ffa1c0a2770 0x7ffa1c0a0b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45436/0 (socket says 172.21.15.6:45436) 2024-04-25T21:40:04.860 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1b7fe640 1 -- 172.21.15.6:0/3563780435 learned_addr learned my addr 172.21.15.6:0/3563780435 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:04.863 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1b7fe640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6790/0 conn(0x7ffa1c0a2770 legacy=0x7ffa1c0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:04.863 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1bfff640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a16b0 legacy=0x7ffa1c13da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:04.863 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa13fff640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.203:6789/0 conn(0x7ffa1c0a1270 legacy=0x7ffa1c13a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:04.863 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121113426 0 0) 0x7ffa1c099820 con 0x7ffa1c0a2770 2024-04-25T21:40:04.863 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:04.863 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:04.863 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): my global_id is 4205 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa08003530 con 0x7ffa1c0a2770 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4054963464 0 0) 0x7ffa1c09a690 con 0x7ffa1c0a16b0 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 10 monclient(hunting): my global_id is 4221 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.856+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa1c099820 con 0x7ffa1c0a16b0 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1bfff640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a16b0 legacy=0x7ffa1c13da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1bfff640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a16b0 legacy=0x7ffa1c13da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1bfff640 1 --1- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a16b0 0x7ffa1c13da20 :-1 s=OPENED pgs=67 cs=1 l=1).handle_message read tag failed 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1bfff640 1 --1- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a16b0 0x7ffa1c13da20 :-1 s=OPENED pgs=67 cs=1 l=1).fault on lossy channel, failing 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-04-25T21:40:04.864 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543116634 0 0) 0x7ffa1c09eef0 con 0x7ffa1c0a1270 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient(hunting): my global_id is 4183 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa1c09a690 con 0x7ffa1c0a1270 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2826646991 0 0) 0x7ffa08003530 con 0x7ffa1c0a2770 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.203:6789/0 conn(0x7ffa1c0a1270 legacy=0x7ffa1c13a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a16b0 legacy=0x7ffa1c13da20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 monclient: found mon.noname-b 2024-04-25T21:40:04.865 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c099df0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: _finish_auth 0 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ffa0c0018d0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: dump: 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:04.866 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 5 monclient: authenticate success, global_id 4205 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffa0c001c20 con 0x7ffa1c0a2770 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 4 set_mon_vals no callback set 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 1 -- 172.21.15.6:0/3563780435 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ffa0c004a50 con 0x7ffa1c0a2770 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1affd640 10 monclient: dump: 2024-04-25T21:40:04.867 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient: shutdown 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- 172.21.15.6:0/3563780435 >> v1:172.21.15.6:6790/0 conn(0x7ffa1c0a2770 legacy=0x7ffa1c0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- 172.21.15.6:0/3563780435 shutdown_connections 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- 172.21.15.6:0/3563780435 >> 172.21.15.6:0/3563780435 conn(0x7ffa1c0a1c50 msgr2=0x7ffa1c0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:04.868 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- 172.21.15.6:0/3563780435 shutdown_connections 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- 172.21.15.6:0/3563780435 wait complete. 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient: build_initial_monmap 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient: monmap: 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 Processor -- start 2024-04-25T21:40:04.869 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- start start 2024-04-25T21:40:04.870 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient: init 2024-04-25T21:40:04.873 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient(hunting): picked mon.noname-c con 0x7ffa1c0a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient(hunting): picked mon.noname-b con 0x7ffa1c0a16b0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient(hunting): picked mon.noname-a con 0x7ffa1c0a2770 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c13d4d0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c13bf60 con 0x7ffa1c0a16b0 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa1c0a4ca0 con 0x7ffa1c0a1270 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa2214c640 10 monclient(hunting): authenticate will time out at 963.774659s 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a2770 0x7ffa1c1d48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47364/0 (socket says 172.21.15.6:47364) 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.860+0000 7ffa1bfff640 1 -- 172.21.15.6:0/3716962451 learned_addr learned my addr 172.21.15.6:0/3716962451 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa1bfff640 1 -- 172.21.15.6:0/3716962451 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a2770 legacy=0x7ffa1c1d48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:04.874 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa1b7fe640 1 -- 172.21.15.6:0/3716962451 >> v1:172.21.15.6:6790/0 conn(0x7ffa1c0a1270 legacy=0x7ffa1c1cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482075822 0 0) 0x7ffa1c13d4d0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): my global_id is 4224 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9fc003530 con 0x7ffa1c0a2770 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700570947 0 0) 0x7ffa1c0a4ca0 con 0x7ffa1c0a1270 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): my global_id is 4208 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa1c13d4d0 con 0x7ffa1c0a1270 2024-04-25T21:40:04.875 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa13fff640 1 -- 172.21.15.6:0/3716962451 >> v1:172.21.15.203:6789/0 conn(0x7ffa1c0a16b0 legacy=0x7ffa1c1d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2883551113 0 0) 0x7ff9fc003530 con 0x7ffa1c0a2770 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa1c0a4ca0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ffa140039a0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:04.876 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3697197523 0 0) 0x7ffa1c13d4d0 con 0x7ffa1c0a1270 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff9fc003530 con 0x7ffa1c0a1270 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ffa0c004990 con 0x7ffa1c0a1270 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:04.877 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1293127858 0 0) 0x7ffa1c0a4ca0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 >> v1:172.21.15.6:6790/0 conn(0x7ffa1c0a1270 legacy=0x7ffa1c1cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 >> v1:172.21.15.203:6789/0 conn(0x7ffa1c0a16b0 legacy=0x7ffa1c1d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 monclient: found mon.a 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c09e1a0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:04.878 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 5 monclient: authenticate success, global_id 4224 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 10 monclient: _renew_subs 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c136140 con 0x7ffa1c0a2770 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 10 monclient: _renew_subs 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffa1c09e5f0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffa14003f40 con 0x7ffa1c0a2770 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ffa14004290 con 0x7ffa1c0a2770 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 10 monclient: dump: 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2024-04-25T21:40:04.879 INFO:teuthology.misc.health.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr: 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa1a7fc640 4 set_mon_vals no callback set 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa00005090 con 0x7ffa1c0a2770 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7ffa140044e0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.880 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(16..16 src has 1..16) ==== 3153+0+0 (unknown 3679573346 0 0) 0x7ffa14044860 con 0x7ffa1c0a2770 2024-04-25T21:40:04.881 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.864+0000 7ffa1b7fe640 10 monclient: get_auth_request con 0x7ff9fc0369d0 auth_method 0 2024-04-25T21:40:04.881 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.868+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7ffa14015da0 con 0x7ffa1c0a2770 2024-04-25T21:40:04.881 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.872+0000 7ffa18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:04.881 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:04.872+0000 7ffa18ff9640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:05.037 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.032+0000 7ffa2214c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:05.037 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.032+0000 7ffa2214c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:05.037 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.032+0000 7ffa2214c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:05.037 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.032+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa00005380 con 0x7ffa1c0a2770 2024-04-25T21:40:05.038 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa18ff9640 1 -- 172.21.15.6:0/3716962451 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+9001 (unknown 1054498076 0 3518773920) 0x7ffa140156a0 con 0x7ffa1c0a2770 2024-04-25T21:40:05.038 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:05.038 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa18ff9640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:05.039 INFO:teuthology.misc.health.smithi006.stdout: 2024-04-25T21:40:05.039 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":16,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:40:02.342546+0000","last_up_change":"2024-04-25T21:39:58.313547+0000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T21:39:59.601307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":763650169}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":763650169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":763650169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":763650169}]},"public_addr":"172.21.15.6:6804/763650169","cluster_addr":"172.21.15.6:6805/763650169","heartbeat_back_addr":"172.21.15.6:6809/763650169","heartbeat_front_addr":"172.21.15.6:6807/763650169","state":["exists","up"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":2126016572}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":2126016572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":2126016572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":2126016572}]},"public_addr":"172.21.15.6:6806/2126016572","cluster_addr":"172.21.15.6:6808/2126016572","heartbeat_back_addr":"172.21.15.6:6811/2126016572","heartbeat_front_addr":"172.21.15.6:6810/2126016572","state":["exists","up"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":3408092622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":3408092622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":3408092622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":3408092622}]},"public_addr":"172.21.15.6:6800/3408092622","cluster_addr":"172.21.15.6:6801/3408092622","heartbeat_back_addr":"172.21.15.6:6803/3408092622","heartbeat_front_addr":"172.21.15.6:6802/3408092622","state":["exists","up"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3201437079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3201437079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":3201437079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":3201437079}]},"public_addr":"172.21.15.203:6800/3201437079","cluster_addr":"172.21.15.203:6802/3201437079","heartbeat_back_addr":"172.21.15.203:6806/3201437079","heartbeat_front_addr":"172.21.15.203:6804/3201437079","state":["exists","up"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":1405305648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":1405305648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1405305648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1405305648}]},"public_addr":"172.21.15.203:6801/1405305648","cluster_addr":"172.21.15.203:6803/1405305648","heartbeat_back_addr":"172.21.15.203:6807/1405305648","heartbeat_front_addr":"172.21.15.203:6805/1405305648","state":["exists","up"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1687281622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1687281622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1687281622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1687281622}]},"public_addr":"172.21.15.203:6808/1687281622","cluster_addr":"172.21.15.203:6809/1687281622","heartbeat_back_addr":"172.21.15.203:6811/1687281622","heartbeat_front_addr":"172.21.15.203:6810/1687281622","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.481868+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.631515+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.371692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.722323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.712123+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.636554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 >> v1:172.21.15.203:6812/3164820986 conn(0x7ff9fc0369d0 legacy=0x7ff9fc038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 10 monclient: shutdown 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 >> v1:172.21.15.6:6789/0 conn(0x7ffa1c0a2770 legacy=0x7ffa1c1d48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 shutdown_connections 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 >> 172.21.15.6:0/3716962451 conn(0x7ffa1c0a1c50 msgr2=0x7ffa1c06e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 shutdown_connections 2024-04-25T21:40:05.041 INFO:teuthology.misc.health.smithi006.stderr:2024-04-25T21:40:05.036+0000 7ffa2214c640 1 -- 172.21.15.6:0/3716962451 wait complete. 2024-04-25T21:40:05.055 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-25T21:40:05.055 INFO:tasks.ceph:Creating RBD pool 2024-04-25T21:40:05.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-25T21:40:05.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient: get_monmap_and_config 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient: build_initial_monmap 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient: monmap: 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:05.179792+0000 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:05.179792+0000 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:05.178 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 1 Processor -- start 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 1 -- start start 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient: init 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa40a1b70 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa40a2320 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa413d320 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa406c030 con 0x7f7fa40a1b70 2024-04-25T21:40:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa409a8c0 con 0x7f7fa40a2320 2024-04-25T21:40:05.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa4142e50 con 0x7f7fa413d320 2024-04-25T21:40:05.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:05.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fab453640 10 monclient(hunting): authenticate will time out at 964.089760s 2024-04-25T21:40:05.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fa99f0640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7fa413d320 0x7f7fa413f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45446/0 (socket says 172.21.15.6:45446) 2024-04-25T21:40:05.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fa99f0640 1 -- 172.21.15.6:0/721406188 learned_addr learned my addr 172.21.15.6:0/721406188 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:05.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.176+0000 7f7fa91ef640 1 --1- 172.21.15.6:0/721406188 >> v1:172.21.15.6:6789/0 conn(0x7f7fa40a1b70 0x7f7fa406b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47374/0 (socket says 172.21.15.6:47374) 2024-04-25T21:40:05.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fa91ef640 1 -- 172.21.15.6:0/721406188 >> v1:172.21.15.6:6789/0 conn(0x7f7fa40a1b70 legacy=0x7f7fa406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fa99f0640 1 -- 172.21.15.6:0/721406188 >> v1:172.21.15.6:6790/0 conn(0x7f7fa413d320 legacy=0x7f7fa413f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2422650938 0 0) 0x7f7fa406c030 con 0x7f7fa40a1b70 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): my global_id is 4227 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f90003530 con 0x7f7fa40a1b70 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473944263 0 0) 0x7f7fa4142e50 con 0x7f7fa413d320 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient(hunting): my global_id is 4211 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fa406c030 con 0x7f7fa413d320 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fa89ee640 1 -- 172.21.15.6:0/721406188 >> v1:172.21.15.203:6789/0 conn(0x7f7fa40a2320 legacy=0x7f7fa406b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1018093555 0 0) 0x7f7f90003530 con 0x7f7fa40a1b70 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 >> v1:172.21.15.6:6790/0 conn(0x7f7fa413d320 legacy=0x7f7fa413f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 >> v1:172.21.15.203:6789/0 conn(0x7f7fa40a2320 legacy=0x7f7fa406b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 monclient: found mon.noname-a 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa4144030 con 0x7f7fa40a1b70 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: _finish_auth 0 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7f9c001c50 con 0x7f7fa40a1b70 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:05.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: dump: 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 5 monclient: authenticate success, global_id 4227 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f9c001fa0 con 0x7f7fa40a1b70 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:05.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 1 -- 172.21.15.6:0/721406188 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7f9c004d10 con 0x7f7fa40a1b70 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7f9bfff640 10 monclient: dump: 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:05.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 4 set_mon_vals no callback set 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient: shutdown 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- 172.21.15.6:0/721406188 >> v1:172.21.15.6:6789/0 conn(0x7f7fa40a1b70 legacy=0x7f7fa406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- 172.21.15.6:0/721406188 shutdown_connections 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- 172.21.15.6:0/721406188 >> 172.21.15.6:0/721406188 conn(0x7f7fa40a2e30 msgr2=0x7f7fa40a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- 172.21.15.6:0/721406188 shutdown_connections 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- 172.21.15.6:0/721406188 wait complete. 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient: build_initial_monmap 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient: monmap: 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:05.189 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 Processor -- start 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- start start 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient: init 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa40a1b70 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa40a2320 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa413d320 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa4142570 con 0x7f7fa413d320 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa40722f0 con 0x7f7fa40a1b70 2024-04-25T21:40:05.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa413e7b0 con 0x7f7fa40a2320 2024-04-25T21:40:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fab453640 10 monclient(hunting): authenticate will time out at 964.094997s 2024-04-25T21:40:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fa89ee640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7fa40a2320 0x7f7fa41d1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45458/0 (socket says 172.21.15.6:45458) 2024-04-25T21:40:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fa89ee640 1 -- 172.21.15.6:0/2875918916 learned_addr learned my addr 172.21.15.6:0/2875918916 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:05.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.180+0000 7f7fa99f0640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7fa413d320 0x7f7fa41d4ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47384/0 (socket says 172.21.15.6:47384) 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fa89ee640 1 -- 172.21.15.6:0/2875918916 >> v1:172.21.15.6:6790/0 conn(0x7f7fa40a2320 legacy=0x7f7fa41d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fa99f0640 1 -- 172.21.15.6:0/2875918916 >> v1:172.21.15.6:6789/0 conn(0x7f7fa413d320 legacy=0x7f7fa41d4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3895670763 0 0) 0x7f7fa0001c10 con 0x7f7fa413d320 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): my global_id is 4230 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f88003530 con 0x7f7fa413d320 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632483310 0 0) 0x7f7fa413e7b0 con 0x7f7fa40a2320 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): my global_id is 4214 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fa0001c10 con 0x7f7fa40a2320 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fa91ef640 1 -- 172.21.15.6:0/2875918916 >> v1:172.21.15.203:6789/0 conn(0x7f7fa40a1b70 legacy=0x7f7fa40986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3172944659 0 0) 0x7f7f88003530 con 0x7f7fa413d320 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7fa413e7b0 con 0x7f7fa413d320 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7fa00047f0 con 0x7f7fa413d320 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): dump: 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.195 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3436492620 0 0) 0x7f7fa0001c10 con 0x7f7fa40a2320 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f88003530 con 0x7f7fa40a2320 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7f94001ba0 con 0x7f7fa40a2320 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:05.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): dump: 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782287116 0 0) 0x7f7fa40722f0 con 0x7f7fa40a1b70 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient(hunting): my global_id is 4192 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fa0001c10 con 0x7f7fa40a1b70 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4164763137 0 0) 0x7f7f88003530 con 0x7f7fa40a2320 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 >> v1:172.21.15.203:6789/0 conn(0x7f7fa40a1b70 legacy=0x7f7fa40986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 >> v1:172.21.15.6:6789/0 conn(0x7f7fa413d320 legacy=0x7f7fa41d4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 monclient: found mon.c 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa413e4d0 con 0x7f7fa40a2320 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:05.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 5 monclient: authenticate success, global_id 4214 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 10 monclient: _renew_subs 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa4072bb0 con 0x7f7fa40a2320 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 10 monclient: _renew_subs 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7fa40717a0 con 0x7f7fa40a2320 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f94003950 con 0x7f7fa40a2320 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7f94003ca0 con 0x7f7fa40a2320 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f9b7fe640 4 set_mon_vals no callback set 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 10 monclient: dump: 2024-04-25T21:40:05.199 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f84005090 con 0x7f7fa40a2320 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f7f94003e40 con 0x7f7fa40a2320 2024-04-25T21:40:05.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 3153+0+0 (unknown 3679573346 0 0) 0x7f7f94044250 con 0x7f7fa40a2320 2024-04-25T21:40:05.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.184+0000 7f7fa91ef640 10 monclient: get_auth_request con 0x7f7f88036bf0 auth_method 0 2024-04-25T21:40:05.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.188+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f7f940156d0 con 0x7f7fa40a2320 2024-04-25T21:40:05.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.188+0000 7f7f99ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:05.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.192+0000 7f7f99ffb640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:05.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.352+0000 7f7fab453640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-25T21:40:05.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.352+0000 7f7fab453640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-25T21:40:05.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.352+0000 7f7fab453640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:05.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.352+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f7f84005b90 con 0x7f7fa40a2320 2024-04-25T21:40:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.452+0000 7f7f99ffb640 1 -- 172.21.15.6:0/2875918916 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) ==== 109+0+0 (unknown 1910661782 0 0) 0x7f7f94019380 con 0x7f7fa40a2320 2024-04-25T21:40:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.452+0000 7f7f99ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-25T21:40:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.452+0000 7f7f99ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-25T21:40:05.457 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 >> v1:172.21.15.203:6812/3164820986 conn(0x7f7f88036bf0 legacy=0x7f7f880390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 10 monclient: shutdown 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 >> v1:172.21.15.6:6790/0 conn(0x7f7fa40a2320 legacy=0x7f7fa41d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 shutdown_connections 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 >> 172.21.15.6:0/2875918916 conn(0x7f7fa40a2e30 msgr2=0x7f7fa40a4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 shutdown_connections 2024-04-25T21:40:05.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.456+0000 7f7fab453640 1 -- 172.21.15.6:0/2875918916 wait complete. 2024-04-25T21:40:05.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-25T21:40:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient: get_monmap_and_config 2024-04-25T21:40:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient: build_initial_monmap 2024-04-25T21:40:05.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient: monmap: 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:05.612406+0000 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:05.612406+0000 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:05.611 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 1 Processor -- start 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 1 -- start start 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient: init 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient(hunting): picked mon.noname-b con 0x7f0348072110 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient(hunting): picked mon.noname-c con 0x7f03480724f0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient(hunting): picked mon.noname-a con 0x7f0348072900 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034809f910 con 0x7f0348072900 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034806e2b0 con 0x7f03480724f0 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03480a3220 con 0x7f0348072110 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:05.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f03503a1640 10 monclient(hunting): authenticate will time out at 964.522312s 2024-04-25T21:40:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f034e13d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f0348072110 0x7f0348133840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45468/0 (socket says 172.21.15.6:45468) 2024-04-25T21:40:05.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.608+0000 7f034e13d640 1 -- 172.21.15.6:0/1871173939 learned_addr learned my addr 172.21.15.6:0/1871173939 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:05.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034e13d640 1 -- 172.21.15.6:0/1871173939 >> v1:172.21.15.6:6790/0 conn(0x7f0348072110 legacy=0x7f0348133840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034e93e640 1 -- 172.21.15.6:0/1871173939 >> v1:172.21.15.6:6789/0 conn(0x7f0348072900 legacy=0x7f034813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 415140688 0 0) 0x7f03480a3220 con 0x7f0348072110 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): my global_id is 4217 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0334003530 con 0x7f0348072110 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210610607 0 0) 0x7f034809f910 con 0x7f0348072900 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient(hunting): my global_id is 4233 2024-04-25T21:40:05.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03480a3220 con 0x7f0348072900 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d93c640 1 -- 172.21.15.6:0/1871173939 >> v1:172.21.15.203:6789/0 conn(0x7f03480724f0 legacy=0x7f034813bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 291860229 0 0) 0x7f0334003530 con 0x7f0348072110 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 >> v1:172.21.15.203:6789/0 conn(0x7f03480724f0 legacy=0x7f034813bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 >> v1:172.21.15.6:6789/0 conn(0x7f0348072900 legacy=0x7f034813f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: _finish_hunting 0 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 monclient: found mon.noname-b 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0348143aa0 con 0x7f0348072110 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: _finish_auth 0 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: _check_auth_tickets 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f03440018d0 con 0x7f0348072110 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:05.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: dump: 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 5 monclient: authenticate success, global_id 4217 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0344001c20 con 0x7f0348072110 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 1 -- 172.21.15.6:0/1871173939 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f0344004a50 con 0x7f0348072110 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d13b640 10 monclient: dump: 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.621 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 4 set_mon_vals no callback set 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient: shutdown 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- 172.21.15.6:0/1871173939 >> v1:172.21.15.6:6790/0 conn(0x7f0348072110 legacy=0x7f0348133840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- 172.21.15.6:0/1871173939 shutdown_connections 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- 172.21.15.6:0/1871173939 >> 172.21.15.6:0/1871173939 conn(0x7f03480a2440 msgr2=0x7f03480a2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- 172.21.15.6:0/1871173939 shutdown_connections 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- 172.21.15.6:0/1871173939 wait complete. 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient: build_initial_monmap 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient: monmap: 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 Processor -- start 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- start start 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient: init 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:05.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient(hunting): picked mon.noname-b con 0x7f0348072110 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient(hunting): picked mon.noname-a con 0x7f03480724f0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient(hunting): picked mon.noname-c con 0x7f034813fe70 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0348137ca0 con 0x7f03480724f0 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0348139cb0 con 0x7f0348072110 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0348138580 con 0x7f034813fe70 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f03503a1640 10 monclient(hunting): authenticate will time out at 964.527634s 2024-04-25T21:40:05.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d93c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03480724f0 0x7f03481d0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47406/0 (socket says 172.21.15.6:47406) 2024-04-25T21:40:05.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.612+0000 7f034d93c640 1 -- 172.21.15.6:0/285921068 learned_addr learned my addr 172.21.15.6:0/285921068 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f034d93c640 1 -- 172.21.15.6:0/285921068 >> v1:172.21.15.6:6789/0 conn(0x7f03480724f0 legacy=0x7f03481d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f034e93e640 1 -- 172.21.15.6:0/285921068 >> v1:172.21.15.6:6790/0 conn(0x7f034813fe70 legacy=0x7f03481d4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239967231 0 0) 0x7f0348137ca0 con 0x7f03480724f0 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): my global_id is 4236 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f032c003530 con 0x7f03480724f0 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1258468648 0 0) 0x7f0348138580 con 0x7f034813fe70 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): my global_id is 4220 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0348137ca0 con 0x7f034813fe70 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f034e13d640 1 -- 172.21.15.6:0/285921068 >> v1:172.21.15.203:6789/0 conn(0x7f0348072110 legacy=0x7f034813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 107189683 0 0) 0x7f032c003530 con 0x7f03480724f0 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0348138580 con 0x7f03480724f0 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f0338001c60 con 0x7f03480724f0 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): dump: 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 371845127 0 0) 0x7f0348137ca0 con 0x7f034813fe70 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f032c003530 con 0x7f034813fe70 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f03400039a0 con 0x7f034813fe70 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): dump: 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.630 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400237609 0 0) 0x7f0348139cb0 con 0x7f0348072110 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient(hunting): my global_id is 4198 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0348137ca0 con 0x7f0348072110 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2215479430 0 0) 0x7f0348138580 con 0x7f03480724f0 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 >> v1:172.21.15.6:6790/0 conn(0x7f034813fe70 legacy=0x7f03481d4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 >> v1:172.21.15.203:6789/0 conn(0x7f0348072110 legacy=0x7f034813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:05.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 monclient: found mon.a 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03481c8e90 con 0x7f03480724f0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: _finish_auth 0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 5 monclient: authenticate success, global_id 4236 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 10 monclient: _renew_subs 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03481c9120 con 0x7f03480724f0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 10 monclient: _renew_subs 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0348137230 con 0x7f03480724f0 2024-04-25T21:40:05.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0338003a40 con 0x7f03480724f0 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f0338003d90 con 0x7f03480724f0 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f034c93a640 4 set_mon_vals no callback set 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 10 monclient: dump: 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:05.633 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0330005090 con 0x7f03480724f0 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f0338003fb0 con 0x7f03480724f0 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.616+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(17..17 src has 1..17) ==== 3495+0+0 (unknown 2451735262 0 0) 0x7f03380443e0 con 0x7f03480724f0 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.620+0000 7f034e13d640 10 monclient: get_auth_request con 0x7f032c036bb0 auth_method 0 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.624+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f03380157f0 con 0x7f03480724f0 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.624+0000 7f033effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:05.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.624+0000 7f033effd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.784+0000 7f03503a1640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-25T21:40:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.784+0000 7f03503a1640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-25T21:40:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.784+0000 7f03503a1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:05.784+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f0330005380 con 0x7f03480724f0 2024-04-25T21:40:07.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.108+0000 7f033effd640 1 -- 172.21.15.6:0/285921068 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) ==== 173+0+0 (unknown 47361648 0 0) 0x7f03380150f0 con 0x7f03480724f0 2024-04-25T21:40:07.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.108+0000 7f033effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-25T21:40:07.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.108+0000 7f033effd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-25T21:40:07.111 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 >> v1:172.21.15.203:6812/3164820986 conn(0x7f032c036bb0 legacy=0x7f032c039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 10 monclient: shutdown 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 >> v1:172.21.15.6:6789/0 conn(0x7f03480724f0 legacy=0x7f03481d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 shutdown_connections 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 >> 172.21.15.6:0/285921068 conn(0x7f03480a2440 msgr2=0x7f03481321a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 shutdown_connections 2024-04-25T21:40:07.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.112+0000 7f03503a1640 1 -- 172.21.15.6:0/285921068 wait complete. 2024-04-25T21:40:07.140 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T21:40:07.141 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T21:40:07.141 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T21:40:07.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T21:40:07.142 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T21:40:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.264+0000 7f5ce81cf640 10 monclient: get_monmap_and_config 2024-04-25T21:40:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.264+0000 7f5ce81cf640 10 monclient: build_initial_monmap 2024-04-25T21:40:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.264+0000 7f5ce81cf640 10 monclient: monmap: 2024-04-25T21:40:07.268 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:07.270571+0000 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:07.270571+0000 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 1 Processor -- start 2024-04-25T21:40:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 1 -- start start 2024-04-25T21:40:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient: init 2024-04-25T21:40:07.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce00a0ba0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce0072c90 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce0139400 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce006e960 con 0x7f5ce0139400 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce006fc80 con 0x7f5ce0072c90 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce006edf0 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce81cf640 10 monclient(hunting): authenticate will time out at 966.181113s 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce5f6b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5ce00a0ba0 0x7f5ce00a0f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45488/0 (socket says 172.21.15.6:45488) 2024-04-25T21:40:07.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce5f6b640 1 -- 172.21.15.6:0/3442006695 learned_addr learned my addr 172.21.15.6:0/3442006695 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce5f6b640 1 -- 172.21.15.6:0/3442006695 >> v1:172.21.15.6:6790/0 conn(0x7f5ce00a0ba0 legacy=0x7f5ce00a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce676c640 1 -- 172.21.15.6:0/3442006695 >> v1:172.21.15.6:6789/0 conn(0x7f5ce0139400 legacy=0x7f5ce013b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183554483 0 0) 0x7f5ce006e960 con 0x7f5ce0139400 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce4f69640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce4f69640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce576a640 1 -- 172.21.15.6:0/3442006695 >> v1:172.21.15.203:6789/0 conn(0x7f5ce0072c90 legacy=0x7f5ce00a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce4f69640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.268+0000 7f5ce4f69640 10 monclient(hunting): my global_id is 4239 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ccc003530 con 0x7f5ce0139400 2024-04-25T21:40:07.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3351939794 0 0) 0x7f5ce006edf0 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient(hunting): my global_id is 4223 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce006e960 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1689572910 0 0) 0x7f5ce006e960 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 >> v1:172.21.15.203:6789/0 conn(0x7f5ce0072c90 legacy=0x7f5ce00a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 >> v1:172.21.15.6:6789/0 conn(0x7f5ce0139400 legacy=0x7f5ce013b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: _finish_hunting 0 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 monclient: found mon.noname-b 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce013ff40 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: _finish_auth 0 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: _check_auth_tickets 2024-04-25T21:40:07.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5cdc001870 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: dump: 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 5 monclient: authenticate success, global_id 4223 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5cdc001bc0 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 1 -- 172.21.15.6:0/3442006695 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5cdc004a50 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce4f69640 10 monclient: dump: 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.278 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 4 set_mon_vals no callback set 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient: shutdown 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- 172.21.15.6:0/3442006695 >> v1:172.21.15.6:6790/0 conn(0x7f5ce00a0ba0 legacy=0x7f5ce00a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- 172.21.15.6:0/3442006695 shutdown_connections 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- 172.21.15.6:0/3442006695 >> 172.21.15.6:0/3442006695 conn(0x7f5ce00a6000 msgr2=0x7f5ce00a6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- 172.21.15.6:0/3442006695 shutdown_connections 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- 172.21.15.6:0/3442006695 wait complete. 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient: build_initial_monmap 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient: monmap: 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:07.279 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:07.280 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 Processor -- start 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- start start 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient: init 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce0072c90 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce00a0ba0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce0139400 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce013f790 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce0135b40 con 0x7f5ce0139400 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce00a57f0 con 0x7f5ce0072c90 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce81cf640 10 monclient(hunting): authenticate will time out at 966.186821s 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce5f6b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5ce0072c90 0x7f5ce0135430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45500/0 (socket says 172.21.15.6:45500) 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.272+0000 7f5ce5f6b640 1 -- 172.21.15.6:0/3556520863 learned_addr learned my addr 172.21.15.6:0/3556520863 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce5f6b640 1 -- 172.21.15.6:0/3556520863 >> v1:172.21.15.6:6790/0 conn(0x7f5ce0072c90 legacy=0x7f5ce0135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce576a640 1 -- 172.21.15.6:0/3556520863 >> v1:172.21.15.6:6789/0 conn(0x7f5ce00a0ba0 legacy=0x7f5ce01cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492577684 0 0) 0x7f5ce00a57f0 con 0x7f5ce0072c90 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): my global_id is 4226 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cc4003530 con 0x7f5ce0072c90 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182346801 0 0) 0x7f5ce013f790 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): my global_id is 4242 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce00a57f0 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce676c640 1 -- 172.21.15.6:0/3556520863 >> v1:172.21.15.203:6789/0 conn(0x7f5ce0139400 legacy=0x7f5ce01cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1682359400 0 0) 0x7f5cc4003530 con 0x7f5ce0072c90 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ce013f790 con 0x7f5ce0072c90 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5cdc0047f0 con 0x7f5ce0072c90 2024-04-25T21:40:07.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): dump: 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3758507721 0 0) 0x7f5ce00a57f0 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cc4003530 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5cd0001b40 con 0x7f5ce00a0ba0 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): dump: 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 612488896 0 0) 0x7f5ce0135b40 con 0x7f5ce0139400 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient(hunting): my global_id is 4204 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce00a57f0 con 0x7f5ce0139400 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1276685588 0 0) 0x7f5ce013f790 con 0x7f5ce0072c90 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 >> v1:172.21.15.203:6789/0 conn(0x7f5ce0139400 legacy=0x7f5ce01cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 >> v1:172.21.15.6:6789/0 conn(0x7f5ce00a0ba0 legacy=0x7f5ce01cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 monclient: found mon.c 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce0138c00 con 0x7f5ce0072c90 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:07.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce81cf640 5 monclient: authenticate success, global_id 4226 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce81cf640 10 monclient: _renew_subs 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce81cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce81cf640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce013e200 con 0x7f5ce0072c90 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce81cf640 10 monclient: _renew_subs 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce81cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce81cf640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ce0138600 con 0x7f5ce0072c90 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5cdc005040 con 0x7f5ce0072c90 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5cdc005390 con 0x7f5ce0072c90 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 10 monclient: dump: 2024-04-25T21:40:07.290 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd7fff640 4 set_mon_vals no callback set 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cc3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cc3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cc3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:07.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cbc005090 con 0x7f5ce0072c90 2024-04-25T21:40:07.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f5cdc010130 con 0x7f5ce0072c90 2024-04-25T21:40:07.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(18..18 src has 1..18) ==== 3506+0+0 (unknown 142423447 0 0) 0x7f5cdc044420 con 0x7f5ce0072c90 2024-04-25T21:40:07.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.276+0000 7f5ce576a640 10 monclient: get_auth_request con 0x7f5cc4036de0 auth_method 0 2024-04-25T21:40:07.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.280+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f5cdc015b30 con 0x7f5ce0072c90 2024-04-25T21:40:07.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.280+0000 7f5cd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.280+0000 7f5cd67fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:07.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.484+0000 7f5cc3fff640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-25T21:40:07.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.484+0000 7f5cc3fff640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-25T21:40:07.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.484+0000 7f5cc3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:07.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.484+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f5cbc005380 con 0x7f5ce0072c90 2024-04-25T21:40:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cd67fc640 1 -- 172.21.15.6:0/3556520863 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+84118 (unknown 481731318 0 2027218942) 0x7f5cdc015430 con 0x7f5ce0072c90 2024-04-25T21:40:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-25T21:40:07.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cd67fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:07.491 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 >> v1:172.21.15.203:6812/3164820986 conn(0x7f5cc4036de0 legacy=0x7f5cc40392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 10 monclient: shutdown 2024-04-25T21:40:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 >> v1:172.21.15.6:6790/0 conn(0x7f5ce0072c90 legacy=0x7f5ce0135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 shutdown_connections 2024-04-25T21:40:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 >> 172.21.15.6:0/3556520863 conn(0x7f5ce00a6000 msgr2=0x7f5ce0070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 shutdown_connections 2024-04-25T21:40:07.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.488+0000 7f5cc3fff640 1 -- 172.21.15.6:0/3556520863 wait complete. 2024-04-25T21:40:07.507 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":3164820986}]},"active_addr":"172.21.15.203:6812/3164820986","active_change":"2024-04-25T21:39:51.505612+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1706922950}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1202748831}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":705829972}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":292357593}]}]} 2024-04-25T21:40:07.510 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T21:40:07.510 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T21:40:07.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.628+0000 7f73be1fb640 10 monclient: get_monmap_and_config 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.628+0000 7f73be1fb640 10 monclient: build_initial_monmap 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.628+0000 7f73be1fb640 10 monclient: monmap: 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:07.634992+0000 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:07.634992+0000 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:07.633 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:07.634 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 1 Processor -- start 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 1 -- start start 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient: init 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient(hunting): picked mon.noname-b con 0x7f73b80a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient(hunting): picked mon.noname-c con 0x7f73b80a4970 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient(hunting): picked mon.noname-a con 0x7f73b809bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b809c590 con 0x7f73b809bab0 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b809adf0 con 0x7f73b80a4970 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b809bf20 con 0x7f73b80a3e80 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73be1fb640 10 monclient(hunting): authenticate will time out at 966.545334s 2024-04-25T21:40:07.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73b809bab0 0x7f73b813f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47448/0 (socket says 172.21.15.6:47448) 2024-04-25T21:40:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b77fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f73b80a3e80 0x7f73b80a4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45506/0 (socket says 172.21.15.6:45506) 2024-04-25T21:40:07.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b7fff640 1 -- 172.21.15.6:0/204559274 learned_addr learned my addr 172.21.15.6:0/204559274 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b7fff640 1 -- 172.21.15.6:0/204559274 >> v1:172.21.15.6:6789/0 conn(0x7f73b809bab0 legacy=0x7f73b813f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b77fe640 1 -- 172.21.15.6:0/204559274 >> v1:172.21.15.6:6790/0 conn(0x7f73b80a3e80 legacy=0x7f73b80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912567085 0 0) 0x7f73b809c590 con 0x7f73b809bab0 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.632+0000 7f73b67fc640 10 monclient(hunting): my global_id is 4245 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a0003530 con 0x7f73b809bab0 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b6ffd640 1 -- 172.21.15.6:0/204559274 >> v1:172.21.15.203:6789/0 conn(0x7f73b80a4970 legacy=0x7f73b813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2469518713 0 0) 0x7f73b809bf20 con 0x7f73b80a3e80 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient(hunting): my global_id is 4229 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73b809c590 con 0x7f73b80a3e80 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4029206554 0 0) 0x7f73a0003530 con 0x7f73b809bab0 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 >> v1:172.21.15.6:6790/0 conn(0x7f73b80a3e80 legacy=0x7f73b80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 >> v1:172.21.15.203:6789/0 conn(0x7f73b80a4970 legacy=0x7f73b813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 monclient: found mon.noname-a 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b806eaf0 con 0x7f73b809bab0 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f73b0001c50 con 0x7f73b809bab0 2024-04-25T21:40:07.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: dump: 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 5 monclient: authenticate success, global_id 4245 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:07.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73b0001fa0 con 0x7f73b809bab0 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 1 -- 172.21.15.6:0/204559274 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f73b0004dd0 con 0x7f73b809bab0 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b67fc640 10 monclient: dump: 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.642 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 4 set_mon_vals no callback set 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient: shutdown 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- 172.21.15.6:0/204559274 >> v1:172.21.15.6:6789/0 conn(0x7f73b809bab0 legacy=0x7f73b813f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- 172.21.15.6:0/204559274 shutdown_connections 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- 172.21.15.6:0/204559274 >> 172.21.15.6:0/204559274 conn(0x7f73b80a5f80 msgr2=0x7f73b80a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- 172.21.15.6:0/204559274 shutdown_connections 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- 172.21.15.6:0/204559274 wait complete. 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient: build_initial_monmap 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient: monmap: 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:07.643 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:07.644 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:07.644 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:07.644 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:07.644 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:07.644 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 Processor -- start 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- start start 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient: init 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient(hunting): picked mon.noname-c con 0x7f73b809bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient(hunting): picked mon.noname-a con 0x7f73b80a3e80 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient(hunting): picked mon.noname-b con 0x7f73b80a4970 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b81a8ec0 con 0x7f73b80a3e80 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b81a96c0 con 0x7f73b80a4970 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73b81ab110 con 0x7f73b809bab0 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73be1fb640 10 monclient(hunting): authenticate will time out at 966.550570s 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b77fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f73b809bab0 0x7f73b81cb820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45508/0 (socket says 172.21.15.6:45508) 2024-04-25T21:40:07.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b77fe640 1 -- 172.21.15.6:0/2873986357 learned_addr learned my addr 172.21.15.6:0/2873986357 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.636+0000 7f73b6ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73b80a3e80 0x7f73b81bcfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47464/0 (socket says 172.21.15.6:47464) 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73b77fe640 1 -- 172.21.15.6:0/2873986357 >> v1:172.21.15.6:6790/0 conn(0x7f73b809bab0 legacy=0x7f73b81cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73b6ffd640 1 -- 172.21.15.6:0/2873986357 >> v1:172.21.15.6:6789/0 conn(0x7f73b80a3e80 legacy=0x7f73b81bcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999150988 0 0) 0x7f73b81a8ec0 con 0x7f73b80a3e80 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): my global_id is 4248 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a4003530 con 0x7f73b80a3e80 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867030902 0 0) 0x7f73b81ab110 con 0x7f73b809bab0 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): my global_id is 4232 2024-04-25T21:40:07.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73b81a8ec0 con 0x7f73b809bab0 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73b7fff640 1 -- 172.21.15.6:0/2873986357 >> v1:172.21.15.203:6789/0 conn(0x7f73b80a4970 legacy=0x7f73b81d3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4158165200 0 0) 0x7f73a4003530 con 0x7f73b80a3e80 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73b81ab110 con 0x7f73b80a3e80 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f73ac001b40 con 0x7f73b80a3e80 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): dump: 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.650 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 772257253 0 0) 0x7f73b81a8ec0 con 0x7f73b809bab0 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73a4003530 con 0x7f73b809bab0 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f73a8001c40 con 0x7f73b809bab0 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient(hunting): dump: 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.651 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4262384866 0 0) 0x7f73b81ab110 con 0x7f73b80a3e80 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 >> v1:172.21.15.6:6790/0 conn(0x7f73b809bab0 legacy=0x7f73b81cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 >> v1:172.21.15.203:6789/0 conn(0x7f73b80a4970 legacy=0x7f73b81d3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 monclient: found mon.a 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b81ab3f0 con 0x7f73b80a3e80 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: _finish_auth 0 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:07.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 5 monclient: authenticate success, global_id 4248 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 10 monclient: _renew_subs 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73b81bd6d0 con 0x7f73b80a3e80 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 10 monclient: _renew_subs 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f73b81aa3f0 con 0x7f73b80a3e80 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73ac0038d0 con 0x7f73b80a3e80 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f73ac003c20 con 0x7f73b80a3e80 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 10 monclient: dump: 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.653 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73b5ffb640 4 set_mon_vals no callback set 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7398005090 con 0x7f73b80a3e80 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f73ac003e00 con 0x7f73b80a3e80 2024-04-25T21:40:07.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3506+0+0 (unknown 142423447 0 0) 0x7f73ac044370 con 0x7f73b80a3e80 2024-04-25T21:40:07.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.640+0000 7f73b77fe640 10 monclient: get_auth_request con 0x7f73a4036a50 auth_method 0 2024-04-25T21:40:07.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.644+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f73ac015690 con 0x7f73b80a3e80 2024-04-25T21:40:07.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.644+0000 7f739ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.644+0000 7f739ffff640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.808+0000 7f73be1fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.808+0000 7f73be1fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.808+0000 7f73be1fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.808+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7398005380 con 0x7f73b80a3e80 2024-04-25T21:40:07.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.808+0000 7f739ffff640 1 -- 172.21.15.6:0/2873986357 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10786 (unknown 3338201251 0 1148200836) 0x7f73ac019340 con 0x7f73b80a3e80 2024-04-25T21:40:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.808+0000 7f739ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.808+0000 7f739ffff640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:07.813 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:07.813 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:40:06.452626+0000","last_up_change":"2024-04-25T21:39:58.313547+0000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T21:39:59.601307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T21:40:05.399371+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":763650169}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":763650169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":763650169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":763650169}]},"public_addr":"172.21.15.6:6804/763650169","cluster_addr":"172.21.15.6:6805/763650169","heartbeat_back_addr":"172.21.15.6:6809/763650169","heartbeat_front_addr":"172.21.15.6:6807/763650169","state":["exists","up"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":2126016572}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":2126016572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":2126016572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":2126016572}]},"public_addr":"172.21.15.6:6806/2126016572","cluster_addr":"172.21.15.6:6808/2126016572","heartbeat_back_addr":"172.21.15.6:6811/2126016572","heartbeat_front_addr":"172.21.15.6:6810/2126016572","state":["exists","up"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":3408092622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":3408092622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":3408092622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":3408092622}]},"public_addr":"172.21.15.6:6800/3408092622","cluster_addr":"172.21.15.6:6801/3408092622","heartbeat_back_addr":"172.21.15.6:6803/3408092622","heartbeat_front_addr":"172.21.15.6:6802/3408092622","state":["exists","up"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3201437079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3201437079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":3201437079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":3201437079}]},"public_addr":"172.21.15.203:6800/3201437079","cluster_addr":"172.21.15.203:6802/3201437079","heartbeat_back_addr":"172.21.15.203:6806/3201437079","heartbeat_front_addr":"172.21.15.203:6804/3201437079","state":["exists","up"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":1405305648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":1405305648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1405305648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1405305648}]},"public_addr":"172.21.15.203:6801/1405305648","cluster_addr":"172.21.15.203:6803/1405305648","heartbeat_back_addr":"172.21.15.203:6807/1405305648","heartbeat_front_addr":"172.21.15.203:6805/1405305648","state":["exists","up"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1687281622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1687281622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1687281622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1687281622}]},"public_addr":"172.21.15.203:6808/1687281622","cluster_addr":"172.21.15.203:6809/1687281622","heartbeat_back_addr":"172.21.15.203:6811/1687281622","heartbeat_front_addr":"172.21.15.203:6810/1687281622","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.481868+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.631515+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.371692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.722323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.712123+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.636554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:40:07.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 >> v1:172.21.15.203:6812/3164820986 conn(0x7f73a4036a50 legacy=0x7f73a4038f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 10 monclient: shutdown 2024-04-25T21:40:07.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 >> v1:172.21.15.6:6789/0 conn(0x7f73b80a3e80 legacy=0x7f73b81bcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 shutdown_connections 2024-04-25T21:40:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 >> 172.21.15.6:0/2873986357 conn(0x7f73b80a5f80 msgr2=0x7f73b809ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 shutdown_connections 2024-04-25T21:40:07.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.812+0000 7f73be1fb640 1 -- 172.21.15.6:0/2873986357 wait complete. 2024-04-25T21:40:07.831 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T21:40:07.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient: get_monmap_and_config 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient: build_initial_monmap 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient: monmap: 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:07.958052+0000 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:07.958052+0000 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:07.956 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:07.957 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:07.957 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 1 Processor -- start 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 1 -- start start 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient: init 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient(hunting): picked mon.noname-b con 0x7fc88c0a2c10 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient(hunting): picked mon.noname-a con 0x7fc88c139e50 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient(hunting): picked mon.noname-c con 0x7fc88c13d960 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:07.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0a0e40 con 0x7fc88c139e50 2024-04-25T21:40:07.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0a1400 con 0x7fc88c13d960 2024-04-25T21:40:07.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0a3700 con 0x7fc88c0a2c10 2024-04-25T21:40:07.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:07.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.952+0000 7fc893e48640 10 monclient(hunting): authenticate will time out at 966.867942s 2024-04-25T21:40:07.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc891be4640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc88c0a2c10 0x7fc88c0a2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45522/0 (socket says 172.21.15.6:45522) 2024-04-25T21:40:07.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc891be4640 1 -- 172.21.15.6:0/3272252129 learned_addr learned my addr 172.21.15.6:0/3272252129 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc891be4640 1 -- 172.21.15.6:0/3272252129 >> v1:172.21.15.6:6790/0 conn(0x7fc88c0a2c10 legacy=0x7fc88c0a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc8913e3640 1 -- 172.21.15.6:0/3272252129 >> v1:172.21.15.6:6789/0 conn(0x7fc88c139e50 legacy=0x7fc88c13c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc8923e5640 1 -- 172.21.15.6:0/3272252129 >> v1:172.21.15.203:6789/0 conn(0x7fc88c13d960 legacy=0x7fc88c13fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254767836 0 0) 0x7fc88c0a3700 con 0x7fc88c0a2c10 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): my global_id is 4235 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc878003530 con 0x7fc88c0a2c10 2024-04-25T21:40:07.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2146123342 0 0) 0x7fc88c0a0e40 con 0x7fc88c139e50 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): my global_id is 4251 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc88c0a3700 con 0x7fc88c139e50 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3173456409 0 0) 0x7fc88c0a1400 con 0x7fc88c13d960 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient(hunting): my global_id is 4210 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc88c0a0e40 con 0x7fc88c13d960 2024-04-25T21:40:07.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1883946089 0 0) 0x7fc878003530 con 0x7fc88c0a2c10 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 >> v1:172.21.15.203:6789/0 conn(0x7fc88c13d960 legacy=0x7fc88c13fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 >> v1:172.21.15.6:6789/0 conn(0x7fc88c139e50 legacy=0x7fc88c13c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: _finish_hunting 0 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 monclient: found mon.noname-b 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c06e9f0 con 0x7fc88c0a2c10 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: _finish_auth 0 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: _check_auth_tickets 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc888001c50 con 0x7fc88c0a2c10 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:07.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: dump: 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc87c001920 con 0x7fc88c139e50 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc893e48640 5 monclient: authenticate success, global_id 4235 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc893e48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc888001fa0 con 0x7fc88c0a2c10 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 1 -- 172.21.15.6:0/3272252129 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc888004dd0 con 0x7fc88c0a2c10 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc893e48640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc893e48640 4 set_mon_vals no callback set 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc890be2640 10 monclient: dump: 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.965 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc893e48640 10 monclient: shutdown 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc893e48640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:07.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.956+0000 7fc893e48640 1 -- 172.21.15.6:0/3272252129 >> v1:172.21.15.6:6790/0 conn(0x7fc88c0a2c10 legacy=0x7fc88c0a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- 172.21.15.6:0/3272252129 shutdown_connections 2024-04-25T21:40:07.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- 172.21.15.6:0/3272252129 >> 172.21.15.6:0/3272252129 conn(0x7fc88c0a5e10 msgr2=0x7fc88c0a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- 172.21.15.6:0/3272252129 shutdown_connections 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- 172.21.15.6:0/3272252129 wait complete. 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient: build_initial_monmap 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient: monmap: 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:07.972 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 Processor -- start 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- start start 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient: init 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient(hunting): picked mon.noname-c con 0x7fc88c0a2c10 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient(hunting): picked mon.noname-b con 0x7fc88c139e50 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient(hunting): picked mon.noname-a con 0x7fc88c13d960 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c121070 con 0x7fc88c13d960 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0a5930 con 0x7fc88c139e50 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0a7920 con 0x7fc88c0a2c10 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc893e48640 10 monclient(hunting): authenticate will time out at 966.873659s 2024-04-25T21:40:07.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc891be4640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc88c0a2c10 0x7fc88c1cbb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45530/0 (socket says 172.21.15.6:45530) 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc891be4640 1 -- 172.21.15.6:0/2034077575 learned_addr learned my addr 172.21.15.6:0/2034077575 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc8923e5640 1 -- 172.21.15.6:0/2034077575 >> v1:172.21.15.6:6789/0 conn(0x7fc88c13d960 legacy=0x7fc88c1d40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.960+0000 7fc891be4640 1 -- 172.21.15.6:0/2034077575 >> v1:172.21.15.6:6790/0 conn(0x7fc88c0a2c10 legacy=0x7fc88c1cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2629981893 0 0) 0x7fc88c121070 con 0x7fc88c13d960 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): my global_id is 4254 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc870003530 con 0x7fc88c13d960 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331739207 0 0) 0x7fc88c0a7920 con 0x7fc88c0a2c10 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): my global_id is 4238 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc88c121070 con 0x7fc88c0a2c10 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8913e3640 1 -- 172.21.15.6:0/2034077575 >> v1:172.21.15.203:6789/0 conn(0x7fc88c139e50 legacy=0x7fc88c1cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 898236510 0 0) 0x7fc870003530 con 0x7fc88c13d960 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc88c0a7920 con 0x7fc88c13d960 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc884001b80 con 0x7fc88c13d960 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): dump: 2024-04-25T21:40:07.975 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1985249560 0 0) 0x7fc88c121070 con 0x7fc88c0a2c10 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc870003530 con 0x7fc88c0a2c10 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc888004d10 con 0x7fc88c0a2c10 2024-04-25T21:40:07.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): dump: 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2056503119 0 0) 0x7fc88c0a5930 con 0x7fc88c139e50 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient(hunting): my global_id is 4213 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc88c121070 con 0x7fc88c139e50 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3613918218 0 0) 0x7fc88c0a7920 con 0x7fc88c13d960 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 >> v1:172.21.15.6:6790/0 conn(0x7fc88c0a2c10 legacy=0x7fc88c1cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 >> v1:172.21.15.203:6789/0 conn(0x7fc88c139e50 legacy=0x7fc88c1cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 monclient: found mon.a 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c09f470 con 0x7fc88c13d960 2024-04-25T21:40:07.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 5 monclient: authenticate success, global_id 4254 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 10 monclient: _renew_subs 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c121af0 con 0x7fc88c13d960 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 10 monclient: _renew_subs 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc88c121750 con 0x7fc88c13d960 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc884003950 con 0x7fc88c13d960 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc884003ca0 con 0x7fc88c13d960 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:07.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc883fff640 4 set_mon_vals no callback set 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 10 monclient: dump: 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:07.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc874005090 con 0x7fc88c13d960 2024-04-25T21:40:07.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fc884003e80 con 0x7fc88c13d960 2024-04-25T21:40:07.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(18..18 src has 1..18) ==== 3506+0+0 (unknown 142423447 0 0) 0x7fc884044220 con 0x7fc88c13d960 2024-04-25T21:40:07.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.964+0000 7fc891be4640 10 monclient: get_auth_request con 0x7fc870036d30 auth_method 0 2024-04-25T21:40:07.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.968+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fc8840155a0 con 0x7fc88c13d960 2024-04-25T21:40:07.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.968+0000 7fc8827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:07.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:07.968+0000 7fc8827fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:08.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.132+0000 7fc893e48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:08.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.132+0000 7fc893e48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:08.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.132+0000 7fc893e48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.132+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc874005380 con 0x7fc88c13d960 2024-04-25T21:40:08.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.132+0000 7fc8827fc640 1 -- 172.21.15.6:0/2034077575 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10773 (unknown 941590788 0 124421770) 0x7fc884014ea0 con 0x7fc88c13d960 2024-04-25T21:40:08.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.132+0000 7fc8827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:08.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.132+0000 7fc8827fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:08.137 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:08.137 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:40:08.112324+0000","last_up_change":"2024-04-25T21:39:58.313547+0000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T21:39:59.601307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T21:40:05.399371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":763650169}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":763650169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":763650169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":763650169}]},"public_addr":"172.21.15.6:6804/763650169","cluster_addr":"172.21.15.6:6805/763650169","heartbeat_back_addr":"172.21.15.6:6809/763650169","heartbeat_front_addr":"172.21.15.6:6807/763650169","state":["exists","up"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":2126016572}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":2126016572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":2126016572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":2126016572}]},"public_addr":"172.21.15.6:6806/2126016572","cluster_addr":"172.21.15.6:6808/2126016572","heartbeat_back_addr":"172.21.15.6:6811/2126016572","heartbeat_front_addr":"172.21.15.6:6810/2126016572","state":["exists","up"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":3408092622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":3408092622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":3408092622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":3408092622}]},"public_addr":"172.21.15.6:6800/3408092622","cluster_addr":"172.21.15.6:6801/3408092622","heartbeat_back_addr":"172.21.15.6:6803/3408092622","heartbeat_front_addr":"172.21.15.6:6802/3408092622","state":["exists","up"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3201437079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3201437079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":3201437079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":3201437079}]},"public_addr":"172.21.15.203:6800/3201437079","cluster_addr":"172.21.15.203:6802/3201437079","heartbeat_back_addr":"172.21.15.203:6806/3201437079","heartbeat_front_addr":"172.21.15.203:6804/3201437079","state":["exists","up"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":1405305648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":1405305648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1405305648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1405305648}]},"public_addr":"172.21.15.203:6801/1405305648","cluster_addr":"172.21.15.203:6803/1405305648","heartbeat_back_addr":"172.21.15.203:6807/1405305648","heartbeat_front_addr":"172.21.15.203:6805/1405305648","state":["exists","up"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1687281622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1687281622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1687281622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1687281622}]},"public_addr":"172.21.15.203:6808/1687281622","cluster_addr":"172.21.15.203:6809/1687281622","heartbeat_back_addr":"172.21.15.203:6811/1687281622","heartbeat_front_addr":"172.21.15.203:6810/1687281622","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.481868+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.631515+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.371692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.722323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.712123+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.636554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:40:08.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 >> v1:172.21.15.203:6812/3164820986 conn(0x7fc870036d30 legacy=0x7fc8700391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 10 monclient: shutdown 2024-04-25T21:40:08.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 >> v1:172.21.15.6:6789/0 conn(0x7fc88c13d960 legacy=0x7fc88c1d40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 shutdown_connections 2024-04-25T21:40:08.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 >> 172.21.15.6:0/2034077575 conn(0x7fc88c0a5e10 msgr2=0x7fc88c137c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 shutdown_connections 2024-04-25T21:40:08.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.136+0000 7fc893e48640 1 -- 172.21.15.6:0/2034077575 wait complete. 2024-04-25T21:40:08.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T21:40:08.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T21:40:08.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T21:40:08.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T21:40:08.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T21:40:08.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient: monmap: 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.271733+0000 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.271733+0000 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.271 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.271 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.271 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.271 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 1 Processor -- start 2024-04-25T21:40:08.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 1 -- start start 2024-04-25T21:40:08.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient: init 2024-04-25T21:40:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed406e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed409bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed413d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed409b0d0 con 0x7f4ed413d0a0 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ed9ea2640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4ed413d0a0 0x7f4ed413f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47480/0 (socket says 172.21.15.6:47480) 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ed9ea2640 1 -- 172.21.15.6:0/3463819814 learned_addr learned my addr 172.21.15.6:0/3463819814 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 1 -- 172.21.15.6:0/3463819814 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed40a4ca0 con 0x7f4ed409bab0 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 1 -- 172.21.15.6:0/3463819814 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed406f070 con 0x7f4ed406e890 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4edb905640 10 monclient(hunting): authenticate will time out at 967.181528s 2024-04-25T21:40:08.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f5ff1c5b640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f5ff1c5b640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f5ff1c5b640 10 monclient: monmap: 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.275037+0000 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.275037+0000 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.275 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ed9ea2640 1 -- 172.21.15.6:0/3463819814 >> v1:172.21.15.6:6789/0 conn(0x7f4ed413d0a0 legacy=0x7f4ed413f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ed96a1640 1 -- 172.21.15.6:0/3463819814 >> v1:172.21.15.6:6790/0 conn(0x7f4ed406e890 legacy=0x7f4ed40a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ed8ea0640 1 -- 172.21.15.6:0/3463819814 >> v1:172.21.15.203:6789/0 conn(0x7f4ed409bab0 legacy=0x7f4ed40a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 991401074 0 0) 0x7f4ecc0017e0 con 0x7f4ed413d0a0 2024-04-25T21:40:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ecbfff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ecbfff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.268+0000 7f4ecbfff640 10 monclient(hunting): my global_id is 4257 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ec0003530 con 0x7f4ed413d0a0 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 116789251 0 0) 0x7f4ed406f070 con 0x7f4ed406e890 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): my global_id is 4241 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ecc0017e0 con 0x7f4ed406e890 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 226294577 0 0) 0x7f4ed40a4ca0 con 0x7f4ed409bab0 2024-04-25T21:40:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient(hunting): my global_id is 4216 2024-04-25T21:40:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed406f070 con 0x7f4ed409bab0 2024-04-25T21:40:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1560320588 0 0) 0x7f4ecc0017e0 con 0x7f4ed406e890 2024-04-25T21:40:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 >> v1:172.21.15.203:6789/0 conn(0x7f4ed409bab0 legacy=0x7f4ed40a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 1 Processor -- start 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 >> v1:172.21.15.6:6789/0 conn(0x7f4ed413d0a0 legacy=0x7f4ed413f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 monclient: found mon.noname-b 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed409adf0 con 0x7f4ed406e890 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ed0001920 con 0x7f4ed406e890 2024-04-25T21:40:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: dump: 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 5 monclient: authenticate success, global_id 4241 2024-04-25T21:40:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ed0001bf0 con 0x7f4ed406e890 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 1 -- 172.21.15.6:0/3463819814 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ed00037c0 con 0x7f4ed406e890 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ecbfff640 10 monclient: dump: 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.282 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.282 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 4 set_mon_vals no callback set 2024-04-25T21:40:08.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 1 -- start start 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient: init 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient(hunting): picked mon.noname-c con 0x7f5fec0a1df0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient(hunting): picked mon.noname-b con 0x7f5fec072150 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient(hunting): picked mon.noname-a con 0x7f5fec0a28e0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec070940 con 0x7f5fec0a28e0 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec0a32c0 con 0x7f5fec0a1df0 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec0716f0 con 0x7f5fec072150 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5ff1c5b640 10 monclient(hunting): authenticate will time out at 967.185075s 2024-04-25T21:40:08.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5fe3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5fec072150 0x7f5fec13a070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45540/0 (socket says 172.21.15.6:45540) 2024-04-25T21:40:08.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5fe3fff640 1 -- 172.21.15.6:0/3937543129 learned_addr learned my addr 172.21.15.6:0/3937543129 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient: shutdown 2024-04-25T21:40:08.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- 172.21.15.6:0/3463819814 >> v1:172.21.15.6:6790/0 conn(0x7f4ed406e890 legacy=0x7f4ed40a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- 172.21.15.6:0/3463819814 shutdown_connections 2024-04-25T21:40:08.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- 172.21.15.6:0/3463819814 >> 172.21.15.6:0/3463819814 conn(0x7f4ed40a5f80 msgr2=0x7f4ed40a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- 172.21.15.6:0/3463819814 shutdown_connections 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- 172.21.15.6:0/3463819814 wait complete. 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient: monmap: 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.289 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 Processor -- start 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- start start 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient: init 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed406e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed409bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed413d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed4124730 con 0x7f4ed413d0a0 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed41c5bc0 con 0x7f4ed409bab0 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed409c490 con 0x7f4ed406e890 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4edb905640 10 monclient(hunting): authenticate will time out at 967.187157s 2024-04-25T21:40:08.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ed96a1640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4ed406e890 0x7f4ed41cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45548/0 (socket says 172.21.15.6:45548) 2024-04-25T21:40:08.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f4ed96a1640 1 -- 172.21.15.6:0/2682057264 learned_addr learned my addr 172.21.15.6:0/2682057264 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5fe3fff640 1 -- 172.21.15.6:0/3937543129 >> v1:172.21.15.6:6790/0 conn(0x7f5fec072150 legacy=0x7f5fec13a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5febfff640 1 -- 172.21.15.6:0/3937543129 >> v1:172.21.15.6:6789/0 conn(0x7f5fec0a28e0 legacy=0x7f5fec13d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feb7fe640 1 -- 172.21.15.6:0/3937543129 >> v1:172.21.15.203:6789/0 conn(0x7f5fec0a1df0 legacy=0x7f5fec0a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1063454952 0 0) 0x7f5fec0716f0 con 0x7f5fec072150 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): my global_id is 4244 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd8003530 con 0x7f5fec072150 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240499070 0 0) 0x7f5fec070940 con 0x7f5fec0a28e0 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 10 monclient(hunting): my global_id is 4260 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.272+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec0716f0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2624892356 0 0) 0x7f5fec0a32c0 con 0x7f5fec0a1df0 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient(hunting): my global_id is 4219 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec070940 con 0x7f5fec0a1df0 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2217225301 0 0) 0x7f5fec0716f0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 >> v1:172.21.15.6:6790/0 conn(0x7f5fec072150 legacy=0x7f5fec13a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 >> v1:172.21.15.203:6789/0 conn(0x7f5fec0a1df0 legacy=0x7f5fec0a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 monclient: found mon.noname-a 2024-04-25T21:40:08.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec0a3730 con 0x7f5fec0a28e0 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fe40019e0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 5 monclient: authenticate success, global_id 4260 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: dump: 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.295 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fe4001cd0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 1 -- 172.21.15.6:0/3937543129 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fe40037c0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 4 set_mon_vals no callback set 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feaffd640 10 monclient: dump: 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.297 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient: shutdown 2024-04-25T21:40:08.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/3937543129 >> v1:172.21.15.6:6789/0 conn(0x7f5fec0a28e0 legacy=0x7f5fec13d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/3937543129 shutdown_connections 2024-04-25T21:40:08.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/3937543129 >> 172.21.15.6:0/3937543129 conn(0x7f5fec0a73f0 msgr2=0x7f5fec0a77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/3937543129 shutdown_connections 2024-04-25T21:40:08.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/3937543129 wait complete. 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ed96a1640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.6:6790/0 conn(0x7f4ed406e890 legacy=0x7f4ed41cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ed9ea2640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.6:6789/0 conn(0x7f4ed413d0a0 legacy=0x7f4ed41d4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 702843224 0 0) 0x7f4ed409c490 con 0x7f4ed406e890 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): my global_id is 4247 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4eb8003530 con 0x7f4ed406e890 2024-04-25T21:40:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient: monmap: 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.300 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.301 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862991899 0 0) 0x7f4ed4124730 con 0x7f4ed413d0a0 2024-04-25T21:40:08.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): my global_id is 4263 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ed409c490 con 0x7f4ed413d0a0 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1946332969 0 0) 0x7f4eb8003530 con 0x7f4ed406e890 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ed4124730 con 0x7f4ed406e890 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ed00037c0 con 0x7f4ed406e890 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): dump: 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.303 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.304 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.304 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ed8ea0640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.203:6789/0 conn(0x7f4ed409bab0 legacy=0x7f4ed41d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 Processor -- start 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 119595900 0 0) 0x7f4ed409c490 con 0x7f4ed413d0a0 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4eb8003530 con 0x7f4ed413d0a0 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ecc0016d0 con 0x7f4ed413d0a0 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient(hunting): dump: 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3958298609 0 0) 0x7f4ed4124730 con 0x7f4ed406e890 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.203:6789/0 conn(0x7f4ed409bab0 legacy=0x7f4ed41d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.6:6789/0 conn(0x7f4ed413d0a0 legacy=0x7f4ed41d4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 monclient: found mon.c 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed40a91c0 con 0x7f4ed406e890 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 5 monclient: authenticate success, global_id 4247 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 10 monclient: _renew_subs 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed41223d0 con 0x7f4ed406e890 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- start start 2024-04-25T21:40:08.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient: init 2024-04-25T21:40:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 10 monclient: _renew_subs 2024-04-25T21:40:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ed41cc980 con 0x7f4ed406e890 2024-04-25T21:40:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ed0001b70 con 0x7f4ed406e890 2024-04-25T21:40:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4ed4062e50 con 0x7f4ed406e890 2024-04-25T21:40:08.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:08.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient(hunting): picked mon.noname-a con 0x7f5fec072150 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient(hunting): picked mon.noname-b con 0x7f5fec0a1df0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient(hunting): picked mon.noname-c con 0x7f5fec0a28e0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec1c61e0 con 0x7f5fec072150 2024-04-25T21:40:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec13bbd0 con 0x7f5fec0a1df0 2024-04-25T21:40:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fec13dec0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5ff1c5b640 10 monclient(hunting): authenticate will time out at 967.191314s 2024-04-25T21:40:08.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5fec072150 0x7f5fec1c6630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47498/0 (socket says 172.21.15.6:47498) 2024-04-25T21:40:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f5feb7fe640 1 -- 172.21.15.6:0/287061955 learned_addr learned my addr 172.21.15.6:0/287061955 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ed0003f60 con 0x7f4ed406e890 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ec9ffb640 10 monclient: dump: 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.276+0000 7f4ecb7fe640 4 set_mon_vals no callback set 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f4ed0004100 con 0x7f4ed406e890 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f4ed0044390 con 0x7f4ed406e890 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6800/3408092622 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4eb8041340 con 0x7f4eb803d6d0 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f4ed0001d20 con 0x7f4ed406e890 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f4ec9ffb640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f4ed9ea2640 10 monclient: get_auth_request con 0x7f4eb803d6d0 auth_method 0 2024-04-25T21:40:08.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f4ed8ea0640 10 monclient: get_auth_request con 0x7f4eb8036af0 auth_method 0 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5feb7fe640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.6:6789/0 conn(0x7f5fec072150 legacy=0x7f5fec1c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793134478 0 0) 0x7f5fec1c61e0 con 0x7f5fec072150 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): my global_id is 4266 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd8003170 con 0x7f5fec072150 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe3fff640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.203:6789/0 conn(0x7f5fec0a1df0 legacy=0x7f5fec1ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5feb7fe640 1 --1- 172.21.15.6:0/287061955 >> v1:172.21.15.6:6789/0 conn(0x7f5fec072150 0x7f5fec1c6630 :-1 s=OPENED pgs=85 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f5fd8003170 auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2501737467 0 0) 0x7f5fd8003170 con 0x7f5fec072150 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec1c61e0 con 0x7f5fec072150 2024-04-25T21:40:08.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fdc001f40 con 0x7f5fec072150 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.317 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5febfff640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.6:6790/0 conn(0x7f5fec0a28e0 legacy=0x7f5fec1d2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200112848 0 0) 0x7f5fec13bbd0 con 0x7f5fec0a1df0 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): my global_id is 4225 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd8003170 con 0x7f5fec0a1df0 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1491816503 0 0) 0x7f5fec13dec0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): my global_id is 4250 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec13bbd0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2509967776 0 0) 0x7f5fec13bbd0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec13dec0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1080006901 0 0) 0x7f5fd8003170 con 0x7f5fec0a1df0 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec13bbd0 con 0x7f5fec0a1df0 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fe40019a0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:08.319 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fd4001ab0 con 0x7f5fec0a1df0 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.320 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1039374071 0 0) 0x7f5fec13dec0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.203:6789/0 conn(0x7f5fec0a1df0 legacy=0x7f5fec1ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.6:6789/0 conn(0x7f5fec072150 legacy=0x7f5fec1c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 monclient: found mon.c 2024-04-25T21:40:08.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fec13d620 con 0x7f5fec0a28e0 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 730190461 0 0) 0x7f5fec1c61e0 con 0x7f5fec072150 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5fe8ff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 5 monclient: authenticate success, global_id 4250 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 10 monclient: _renew_subs 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5fec13d900 con 0x7f5fec0a28e0 2024-04-25T21:40:08.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 10 monclient: _renew_subs 2024-04-25T21:40:08.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5fec13cab0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.280+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5fec062e50 con 0x7f5fec0a28e0 2024-04-25T21:40:08.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fe4001cd0 con 0x7f5fec0a28e0 2024-04-25T21:40:08.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fe4003f80 con 0x7f5fec0a28e0 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe9ffb640 4 set_mon_vals no callback set 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 10 monclient: dump: 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.326 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f5fe4004120 con 0x7f5fec0a28e0 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f5fe4043490 con 0x7f5fec0a28e0 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6804/763650169 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fd8041be0 con 0x7f5fd803df70 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f5fe4003f80 con 0x7f5fec0a28e0 2024-04-25T21:40:08.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe8ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-04-25T21:40:08.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5fe3fff640 10 monclient: get_auth_request con 0x7f5fd803df70 auth_method 0 2024-04-25T21:40:08.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f5feb7fe640 10 monclient: get_auth_request con 0x7f5fd8037320 auth_method 0 2024-04-25T21:40:08.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.284+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== osd.2 v1:172.21.15.6:6800/3408092622 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4eb8041340 con 0x7f4eb803d6d0 2024-04-25T21:40:08.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.288+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== osd.0 v1:172.21.15.6:6804/763650169 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5fd8041be0 con 0x7f5fd803df70 2024-04-25T21:40:08.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.308+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 --> v1:172.21.15.6:6800/3408092622 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4ed413c3e0 con 0x7f4eb803d6d0 2024-04-25T21:40:08.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4ec9ffb640 1 -- 172.21.15.6:0/2682057264 <== osd.2 v1:172.21.15.6:6800/3408092622 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 839772338) 0x7f4ed413c3e0 con 0x7f4eb803d6d0 2024-04-25T21:40:08.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.6:6800/3408092622 conn(0x7f4eb803d6d0 legacy=0x7f4eb803faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.203:6812/3164820986 conn(0x7f4eb8036af0 legacy=0x7f4eb8038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 10 monclient: shutdown 2024-04-25T21:40:08.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 >> v1:172.21.15.6:6790/0 conn(0x7f4ed406e890 legacy=0x7f4ed41cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4ed9ea2640 1 -- 172.21.15.6:0/2682057264 reap_dead start 2024-04-25T21:40:08.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 shutdown_connections 2024-04-25T21:40:08.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 >> 172.21.15.6:0/2682057264 conn(0x7f4ed40a5f80 msgr2=0x7f4ed409ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 shutdown_connections 2024-04-25T21:40:08.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.312+0000 7f4edb905640 1 -- 172.21.15.6:0/2682057264 wait complete. 2024-04-25T21:40:08.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.316+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 --> v1:172.21.15.6:6804/763650169 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5fec0a4da0 con 0x7f5fd803df70 2024-04-25T21:40:08.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.316+0000 7f5fe8ff9640 1 -- 172.21.15.6:0/287061955 <== osd.0 v1:172.21.15.6:6804/763650169 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 839772338) 0x7f5fec0a4da0 con 0x7f5fd803df70 2024-04-25T21:40:08.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.6:6804/763650169 conn(0x7f5fd803df70 legacy=0x7f5fd8040390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.203:6812/3164820986 conn(0x7f5fd8037320 legacy=0x7f5fd80397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 10 monclient: shutdown 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 >> v1:172.21.15.6:6790/0 conn(0x7f5fec0a28e0 legacy=0x7f5fec1d2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5febfff640 1 -- 172.21.15.6:0/287061955 reap_dead start 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 shutdown_connections 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 >> 172.21.15.6:0/287061955 conn(0x7f5fec0a73f0 msgr2=0x7f5fec09b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 shutdown_connections 2024-04-25T21:40:08.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f5ff1c5b640 1 -- 172.21.15.6:0/287061955 wait complete. 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient: monmap: 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.325468+0000 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.325468+0000 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 1 Processor -- start 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 1 -- start start 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient: init 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed40b54d0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed40b58b0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed40b5d10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed4023400 con 0x7f0ed40b5d10 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed40236e0 con 0x7f0ed40b58b0 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed40239c0 con 0x7f0ed40b54d0 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edb7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0ed40b5d10 0x7f0ed40c43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47500/0 (socket says 172.21.15.6:47500) 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edbfff640 10 monclient(hunting): authenticate will time out at 967.235374s 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.320+0000 7f0edb7fe640 1 -- 172.21.15.6:0/3672133909 learned_addr learned my addr 172.21.15.6:0/3672133909 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edb7fe640 1 -- 172.21.15.6:0/3672133909 >> v1:172.21.15.6:6789/0 conn(0x7f0ed40b5d10 legacy=0x7f0ed40c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0eda7fc640 1 -- 172.21.15.6:0/3672133909 >> v1:172.21.15.203:6789/0 conn(0x7f0ed40b58b0 legacy=0x7f0ed40c0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388342964 0 0) 0x7f0ed4023400 con 0x7f0ed40b5d10 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): my global_id is 4269 2024-04-25T21:40:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ec4003530 con 0x7f0ed40b5d10 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447065 0 0) 0x7f0ed40236e0 con 0x7f0ed40b58b0 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient(hunting): my global_id is 4228 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed4023400 con 0x7f0ed40b58b0 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1233808541 0 0) 0x7f0ec4003530 con 0x7f0ed40b5d10 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 >> v1:172.21.15.6:6790/0 conn(0x7f0ed40b54d0 legacy=0x7f0ed40246a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 >> v1:172.21.15.203:6789/0 conn(0x7f0ed40b58b0 legacy=0x7f0ed40c0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 monclient: found mon.noname-a 2024-04-25T21:40:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed40265f0 con 0x7f0ed40b5d10 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f0edc0504a0 con 0x7f0ed40b5d10 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: dump: 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 5 monclient: authenticate success, global_id 4269 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0edc067e20 con 0x7f0ed40b5d10 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 1 -- 172.21.15.6:0/3672133909 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f0edc068a90 con 0x7f0ed40b5d10 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0ed9ffb640 10 monclient: dump: 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.342 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 4 set_mon_vals no callback set 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f8a2d18c640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f8a2d18c640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f8a2d18c640 10 monclient: monmap: 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.330518+0000 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.330518+0000 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.345 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f8a2d18c640 1 Processor -- start 2024-04-25T21:40:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f8a2d18c640 1 -- start start 2024-04-25T21:40:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f8a2d18c640 10 monclient: init 2024-04-25T21:40:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 10 monclient: shutdown 2024-04-25T21:40:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 1 -- 172.21.15.6:0/3672133909 >> v1:172.21.15.6:6789/0 conn(0x7f0ed40b5d10 legacy=0x7f0ed40c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 1 -- 172.21.15.6:0/3672133909 shutdown_connections 2024-04-25T21:40:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 1 -- 172.21.15.6:0/3672133909 >> 172.21.15.6:0/3672133909 conn(0x7f0ed402aaf0 msgr2=0x7f0ed402af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 1 -- 172.21.15.6:0/3672133909 shutdown_connections 2024-04-25T21:40:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.324+0000 7f0edbfff640 1 -- 172.21.15.6:0/3672133909 wait complete. 2024-04-25T21:40:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 10 monclient(hunting): picked mon.noname-b con 0x7f8a2806e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 10 monclient(hunting): picked mon.noname-a con 0x7f8a2809bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 10 monclient(hunting): picked mon.noname-c con 0x7f8a2813d0a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2809b0d0 con 0x7f8a2809bab0 2024-04-25T21:40:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a280a4ca0 con 0x7f8a2813d0a0 2024-04-25T21:40:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a26d9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8a2806e890 0x7f8a280a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45578/0 (socket says 172.21.15.6:45578) 2024-04-25T21:40:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a26d9d640 1 -- 172.21.15.6:0/1289158715 learned_addr learned my addr 172.21.15.6:0/1289158715 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1289158715 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2806f070 con 0x7f8a2806e890 2024-04-25T21:40:08.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 10 monclient(hunting): authenticate will time out at 967.240706s 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient: monmap: 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 1 Processor -- start 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 1 -- start start 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient: init 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed40b54d0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed40b58b0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed40b5d10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed40b3b20 con 0x7f0ed40b58b0 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed40b6530 con 0x7f0ed40b5d10 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed40b9520 con 0x7f0ed40b54d0 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0edbfff640 10 monclient(hunting): authenticate will time out at 967.241773s 2024-04-25T21:40:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0eda7fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0ed40b58b0 0x7f0ed41558f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47520/0 (socket says 172.21.15.6:47520) 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f0eda7fc640 1 -- 172.21.15.6:0/304069260 learned_addr learned my addr 172.21.15.6:0/304069260 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient: monmap: 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.334112+0000 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.334112+0000 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.357 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a26d9d640 1 -- 172.21.15.6:0/1289158715 >> v1:172.21.15.6:6790/0 conn(0x7f8a2806e890 legacy=0x7f8a280a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2659c640 1 -- 172.21.15.6:0/1289158715 >> v1:172.21.15.6:6789/0 conn(0x7f8a2809bab0 legacy=0x7f8a280a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2759e640 1 -- 172.21.15.6:0/1289158715 >> v1:172.21.15.203:6789/0 conn(0x7f8a2813d0a0 legacy=0x7f8a2813f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 1 Processor -- start 2024-04-25T21:40:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931080836 0 0) 0x7f8a1c0014b0 con 0x7f8a2809bab0 2024-04-25T21:40:08.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): my global_id is 4272 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a10003530 con 0x7f8a2809bab0 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1371396409 0 0) 0x7f8a2806f070 con 0x7f8a2806e890 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): my global_id is 4253 2024-04-25T21:40:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a1c0014b0 con 0x7f8a2806e890 2024-04-25T21:40:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1447781447 0 0) 0x7f8a280a4ca0 con 0x7f8a2813d0a0 2024-04-25T21:40:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient(hunting): my global_id is 4231 2024-04-25T21:40:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a2806f070 con 0x7f8a2813d0a0 2024-04-25T21:40:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 1 -- start start 2024-04-25T21:40:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient: init 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2797378070 0 0) 0x7f8a1c0014b0 con 0x7f8a2806e890 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 >> v1:172.21.15.203:6789/0 conn(0x7f8a2813d0a0 legacy=0x7f8a2813f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 >> v1:172.21.15.6:6789/0 conn(0x7f8a2809bab0 legacy=0x7f8a280a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 monclient: found mon.noname-b 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2809adf0 con 0x7f8a2806e890 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f8a180018d0 con 0x7f8a2806e890 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a25d9b640 10 monclient: dump: 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient(hunting): picked mon.noname-b con 0x7ff7ec0a1350 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient(hunting): picked mon.noname-c con 0x7ff7ec0a1730 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient(hunting): picked mon.noname-a con 0x7ff7ec139560 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec06fe00 con 0x7ff7ec139560 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec06e7c0 con 0x7ff7ec0a1730 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec0a31c0 con 0x7ff7ec0a1350 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7ff7f1ae7640 10 monclient(hunting): authenticate will time out at 967.243963s 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7eb7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7ff7ec0a1350 0x7ff7ec070df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45590/0 (socket says 172.21.15.6:45590) 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7eb7fe640 1 -- 172.21.15.6:0/4125068974 learned_addr learned my addr 172.21.15.6:0/4125068974 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 5 monclient: authenticate success, global_id 4253 2024-04-25T21:40:08.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.328+0000 7f8a2d18c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0eda7fc640 1 -- 172.21.15.6:0/304069260 >> v1:172.21.15.6:6789/0 conn(0x7f0ed40b58b0 legacy=0x7f0ed41558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edaffd640 1 -- 172.21.15.6:0/304069260 >> v1:172.21.15.6:6790/0 conn(0x7f0ed40b54d0 legacy=0x7f0ed40c4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419032966 0 0) 0x7f0ed40b9520 con 0x7f0ed40b54d0 2024-04-25T21:40:08.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): my global_id is 4256 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ec8003560 con 0x7f0ed40b54d0 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2390393465 0 0) 0x7f0ec8003560 con 0x7f0ed40b54d0 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ed40b9520 con 0x7f0ed40b54d0 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f0ecc004200 con 0x7f0ed40b54d0 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): dump: 2024-04-25T21:40:08.368 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3388109211 0 0) 0x7f0ed40b9520 con 0x7f0ed40b54d0 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 >> v1:172.21.15.203:6789/0 conn(0x7f0ed40b5d10 legacy=0x7f0ed4159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 >> v1:172.21.15.6:6789/0 conn(0x7f0ed40b58b0 legacy=0x7f0ed41558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 monclient: found mon.c 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed40b41e0 con 0x7f0ed40b54d0 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 5 monclient: authenticate success, global_id 4256 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 10 monclient: _renew_subs 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed40ba590 con 0x7f0ed40b54d0 2024-04-25T21:40:08.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a18001c20 con 0x7f8a2806e890 2024-04-25T21:40:08.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a25d9b640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a25d9b640 1 -- 172.21.15.6:0/1289158715 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f8a18004a50 con 0x7f8a2806e890 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 4 set_mon_vals no callback set 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a25d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a25d9b640 10 monclient: dump: 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient: shutdown 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1289158715 >> v1:172.21.15.6:6790/0 conn(0x7f8a2806e890 legacy=0x7f8a280a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1289158715 shutdown_connections 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1289158715 >> 172.21.15.6:0/1289158715 conn(0x7f8a280a5f80 msgr2=0x7f8a280a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1289158715 shutdown_connections 2024-04-25T21:40:08.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1289158715 wait complete. 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 10 monclient: _renew_subs 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ecc001190 con 0x7f0ed40b54d0 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ed40b4d40 con 0x7f0ed40b54d0 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f0ecc0014e0 con 0x7f0ed40b54d0 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 10 monclient: dump: 2024-04-25T21:40:08.374 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ed97fa640 4 set_mon_vals no callback set 2024-04-25T21:40:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient: monmap: 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.376 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7eb7fe640 1 -- 172.21.15.6:0/4125068974 >> v1:172.21.15.6:6790/0 conn(0x7ff7ec0a1350 legacy=0x7ff7ec070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ebfff640 1 -- 172.21.15.6:0/4125068974 >> v1:172.21.15.6:6789/0 conn(0x7ff7ec139560 legacy=0x7ff7ec13b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7eaffd640 1 -- 172.21.15.6:0/4125068974 >> v1:172.21.15.203:6789/0 conn(0x7ff7ec0a1730 legacy=0x7ff7ec0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f0ecc001700 con 0x7f0ed40b54d0 2024-04-25T21:40:08.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f0ecc044370 con 0x7f0ed40b54d0 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2721850720 0 0) 0x7ff7ec0a31c0 con 0x7ff7ec0a1350 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): my global_id is 4259 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7d4005740 con 0x7ff7ec0a1350 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 Processor -- start 2024-04-25T21:40:08.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- start start 2024-04-25T21:40:08.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient: init 2024-04-25T21:40:08.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0eda7fc640 10 monclient: get_auth_request con 0x7f0ec8036b60 auth_method 0 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient(hunting): picked mon.noname-a con 0x7f8a2806e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient(hunting): picked mon.noname-b con 0x7f8a2809bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient(hunting): picked mon.noname-c con 0x7f8a2813d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a280a1910 con 0x7f8a2806e890 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2813ce10 con 0x7f8a2809bab0 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2809c490 con 0x7f8a2813d0a0 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2d18c640 10 monclient(hunting): authenticate will time out at 967.247458s 2024-04-25T21:40:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2759e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f8a2813d0a0 0x7f8a281d83c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45592/0 (socket says 172.21.15.6:45592) 2024-04-25T21:40:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f8a2759e640 1 -- 172.21.15.6:0/1973264231 learned_addr learned my addr 172.21.15.6:0/1973264231 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f4ae50d6640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 --> v1:172.21.15.203:6808/1687281622 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ed4005130 con 0x7f0ed40c5cf0 2024-04-25T21:40:08.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f4ae50d6640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7f4ae50d6640 10 monclient: monmap: 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.338738+0000 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.338738+0000 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.382 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541971484 0 0) 0x7ff7ec06fe00 con 0x7ff7ec139560 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient(hunting): my global_id is 4278 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7ec0a31c0 con 0x7ff7ec139560 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4101776718 0 0) 0x7ff7d4005740 con 0x7ff7ec0a1350 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 >> v1:172.21.15.203:6789/0 conn(0x7ff7ec0a1730 legacy=0x7ff7ec0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 >> v1:172.21.15.6:6789/0 conn(0x7ff7ec139560 legacy=0x7ff7ec13b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 monclient: found mon.noname-b 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec074b10 con 0x7ff7ec0a1350 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ff7dc0018d0 con 0x7ff7ec0a1350 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: dump: 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.384 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7f1ae7640 5 monclient: authenticate success, global_id 4259 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7f1ae7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff7dc001ba0 con 0x7ff7ec0a1350 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 1 -- 172.21.15.6:0/4125068974 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ff7dc0048b0 con 0x7ff7ec0a1350 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.332+0000 7ff7ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7ea7fc640 10 monclient: dump: 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 4 set_mon_vals no callback set 2024-04-25T21:40:08.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f0edb7fe640 10 monclient: get_auth_request con 0x7f0ed40c5cf0 auth_method 0 2024-04-25T21:40:08.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 1 Processor -- start 2024-04-25T21:40:08.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient: shutdown 2024-04-25T21:40:08.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/4125068974 >> v1:172.21.15.6:6790/0 conn(0x7ff7ec0a1350 legacy=0x7ff7ec070df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 1 -- start start 2024-04-25T21:40:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient: init 2024-04-25T21:40:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient(hunting): picked mon.noname-c con 0x7f4ae009eb80 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient(hunting): picked mon.noname-a con 0x7f4ae009ef60 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient(hunting): picked mon.noname-b con 0x7f4ae01396a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae00a06b0 con 0x7f4ae009ef60 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae00a1100 con 0x7f4ae009eb80 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae009cf10 con 0x7f4ae01396a0 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae50d6640 10 monclient(hunting): authenticate will time out at 967.248765s 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/4125068974 shutdown_connections 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/4125068974 >> 172.21.15.6:0/4125068974 conn(0x7ff7ec0a2440 msgr2=0x7ff7ec0a2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/4125068974 shutdown_connections 2024-04-25T21:40:08.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4adf7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4ae009ef60 0x7f4ae009ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47556/0 (socket says 172.21.15.6:47556) 2024-04-25T21:40:08.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4adf7fe640 1 -- 172.21.15.6:0/730254118 learned_addr learned my addr 172.21.15.6:0/730254118 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/4125068974 wait complete. 2024-04-25T21:40:08.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient: monmap: 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.391 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.392 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-04-25T21:40:08.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T21:40:08.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2759e640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.6:6790/0 conn(0x7f8a2813d0a0 legacy=0x7f8a281d83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 Processor -- start 2024-04-25T21:40:08.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a26d9d640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.6:6789/0 conn(0x7f8a2806e890 legacy=0x7f8a280a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2659c640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.203:6789/0 conn(0x7f8a2809bab0 legacy=0x7f8a281d4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1300641636 0 0) 0x7f8a2809c490 con 0x7f8a2813d0a0 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): my global_id is 4262 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a14003530 con 0x7f8a2813d0a0 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 969695474 0 0) 0x7f8a2813ce10 con 0x7f8a2809bab0 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): my global_id is 4237 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a2809c490 con 0x7f8a2809bab0 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4069368232 0 0) 0x7f8a14003530 con 0x7f8a2813d0a0 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8a2813ce10 con 0x7f8a2813d0a0 2024-04-25T21:40:08.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f8a20001fa0 con 0x7f8a2813d0a0 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): dump: 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2427186883 0 0) 0x7f8a2813ce10 con 0x7f8a2813d0a0 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.203:6789/0 conn(0x7f8a2809bab0 legacy=0x7f8a281d4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.6:6789/0 conn(0x7f8a2806e890 legacy=0x7f8a280a30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 monclient: found mon.c 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a280a1630 con 0x7f8a2813d0a0 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- start start 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient: init 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient(hunting): picked mon.noname-b con 0x7ff7ec0a1730 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient(hunting): picked mon.noname-a con 0x7ff7ec139560 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient(hunting): picked mon.noname-c con 0x7ff7ec09d2a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec11c110 con 0x7ff7ec139560 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec09ea90 con 0x7ff7ec0a1730 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ec11c3f0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7f1ae7640 10 monclient(hunting): authenticate will time out at 967.250740s 2024-04-25T21:40:08.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4adf7fe640 1 -- 172.21.15.6:0/730254118 >> v1:172.21.15.6:6789/0 conn(0x7f4ae009ef60 legacy=0x7f4ae009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4ae48d5640 1 -- 172.21.15.6:0/730254118 >> v1:172.21.15.6:6790/0 conn(0x7f4ae01396a0 legacy=0x7f4ae013ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 5 monclient: authenticate success, global_id 4262 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 10 monclient: _renew_subs 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2813f8c0 con 0x7f8a2813d0a0 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 10 monclient: _renew_subs 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8a2813ea30 con 0x7f8a2813d0a0 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8a20003c50 con 0x7f8a2813d0a0 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f8a20003fa0 con 0x7f8a2813d0a0 2024-04-25T21:40:08.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a0f7fe640 10 monclient: dump: 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2559a640 4 set_mon_vals no callback set 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8a08000f90 con 0x7f8a2813d0a0 2024-04-25T21:40:08.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7f4adffff640 1 -- 172.21.15.6:0/730254118 >> v1:172.21.15.203:6789/0 conn(0x7f4ae009eb80 legacy=0x7f4ae009f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7eb7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff7ec0a1730 0x7ff7ec1c7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.6:54378/0 (socket says 172.21.15.6:54378) 2024-04-25T21:40:08.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.336+0000 7ff7eb7fe640 1 -- 172.21.15.6:0/1952468897 learned_addr learned my addr 172.21.15.6:0/1952468897 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 687370841 0 0) 0x7f4ae00a1100 con 0x7f4ae009eb80 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): my global_id is 4240 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ac8003530 con 0x7f4ae009eb80 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1678735066 0 0) 0x7f4ae009cf10 con 0x7f4ae01396a0 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient(hunting): my global_id is 4265 2024-04-25T21:40:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae00a1100 con 0x7f4ae01396a0 2024-04-25T21:40:08.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f8a20004140 con 0x7f8a2813d0a0 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1641752236 0 0) 0x7f4ac8003530 con 0x7f4ae009eb80 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 >> v1:172.21.15.6:6790/0 conn(0x7f4ae01396a0 legacy=0x7f4ae013ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 >> v1:172.21.15.6:6789/0 conn(0x7f4ae009ef60 legacy=0x7f4ae009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 monclient: found mon.noname-c 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae009d1f0 con 0x7f4ae009eb80 2024-04-25T21:40:08.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ad4001920 con 0x7f4ae009eb80 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: dump: 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.406 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.407 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.407 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f8a20044730 con 0x7f8a2813d0a0 2024-04-25T21:40:08.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ad4001c70 con 0x7f4ae009eb80 2024-04-25T21:40:08.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 1 -- 172.21.15.6:0/730254118 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ad40037c0 con 0x7f4ae009eb80 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adeffd640 10 monclient: dump: 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.408 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.203:6801/1405305648 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a14041490 con 0x7f8a1403d820 2024-04-25T21:40:08.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 5 monclient: authenticate success, global_id 4240 2024-04-25T21:40:08.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 4 set_mon_vals no callback set 2024-04-25T21:40:08.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f8a26d9d640 10 monclient: get_auth_request con 0x7f8a14036bd0 auth_method 0 2024-04-25T21:40:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient: shutdown 2024-04-25T21:40:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- 172.21.15.6:0/730254118 >> v1:172.21.15.203:6789/0 conn(0x7f4ae009eb80 legacy=0x7f4ae009f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- 172.21.15.6:0/730254118 shutdown_connections 2024-04-25T21:40:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- 172.21.15.6:0/730254118 >> 172.21.15.6:0/730254118 conn(0x7f4ae00a5710 msgr2=0x7f4ae00a1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- 172.21.15.6:0/730254118 shutdown_connections 2024-04-25T21:40:08.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- 172.21.15.6:0/730254118 wait complete. 2024-04-25T21:40:08.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f8a2659c640 10 monclient: get_auth_request con 0x7f8a1403d820 auth_method 0 2024-04-25T21:40:08.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7eb7fe640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.203:6789/0 conn(0x7ff7ec0a1730 legacy=0x7ff7ec1c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7eaffd640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.6:6789/0 conn(0x7ff7ec139560 legacy=0x7ff7ec1cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201237889 0 0) 0x7ff7ec09ea90 con 0x7ff7ec0a1730 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): my global_id is 4243 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7d8003530 con 0x7ff7ec0a1730 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7eaffd640 1 --1- 172.21.15.6:0/1952468897 >> v1:172.21.15.6:6789/0 conn(0x7ff7ec139560 0x7ff7ec1cc920 :-1 s=OPENED pgs=92 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7ff7ec11c110 auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7ebfff640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.6:6790/0 conn(0x7ff7ec09d2a0 legacy=0x7ff7ec1d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f8a20004480 con 0x7f8a2813d0a0 2024-04-25T21:40:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f8a0f7fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-04-25T21:40:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1257741818 0 0) 0x7ff7ec11c3f0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): my global_id is 4268 2024-04-25T21:40:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7ec09ea90 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient: monmap: 2024-04-25T21:40:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 Processor -- start 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3476603063 0 0) 0x7ff7d8003530 con 0x7ff7ec0a1730 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7ec11c3f0 con 0x7ff7ec0a1730 2024-04-25T21:40:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ff7dc0047f0 con 0x7ff7ec0a1730 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): dump: 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- start start 2024-04-25T21:40:08.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient: init 2024-04-25T21:40:08.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2242401803 0 0) 0x7ff7ec09ea90 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff7d8003530 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ff7e0001bc0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient(hunting): dump: 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== osd.5 v1:172.21.15.203:6808/1687281622 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0ed4005130 con 0x7f0ed40c5cf0 2024-04-25T21:40:08.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4256042619 0 0) 0x7ff7d8003530 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.203:6789/0 conn(0x7ff7ec0a1730 legacy=0x7ff7ec1c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.6:6789/0 conn(0x7ff7ec139560 legacy=0x7ff7ec1cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 monclient: found mon.c 2024-04-25T21:40:08.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ec131ee0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7f1ae7640 5 monclient: authenticate success, global_id 4268 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7f1ae7640 10 monclient: _renew_subs 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7f1ae7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ec09f980 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7f1ae7640 10 monclient: _renew_subs 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7f1ae7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff7ec11c810 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient(hunting): picked mon.noname-c con 0x7f4ae009eb80 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient(hunting): picked mon.noname-a con 0x7f4ae009ef60 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient(hunting): picked mon.noname-b con 0x7f4ae01396a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae0121930 con 0x7f4ae009ef60 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae011fa10 con 0x7f4ae01396a0 2024-04-25T21:40:08.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ae00a0420 con 0x7f4ae009eb80 2024-04-25T21:40:08.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4ae50d6640 10 monclient(hunting): authenticate will time out at 967.255524s 2024-04-25T21:40:08.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adffff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4ae009eb80 0x7f4ae011ebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45628/0 (socket says 172.21.15.6:45628) 2024-04-25T21:40:08.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.340+0000 7f4adffff640 1 -- 172.21.15.6:0/1249852406 learned_addr learned my addr 172.21.15.6:0/1249852406 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.422 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-04-25T21:40:08.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T21:40:08.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7f1ae7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff7ec062e50 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff7e00038d0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7ff7e0003c20 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 10 monclient: dump: 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7e9ffb640 4 set_mon_vals no callback set 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7ff7e0003dc0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7ff7e00441f0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.203:6800/3201437079 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7d8041190 con 0x7ff7d803d5c0 2024-04-25T21:40:08.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7ff7e00446a0 con 0x7ff7ec09d2a0 2024-04-25T21:40:08.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7d3fff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-04-25T21:40:08.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7eb7fe640 10 monclient: get_auth_request con 0x7ff7d8036a10 auth_method 0 2024-04-25T21:40:08.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7ff7eaffd640 10 monclient: get_auth_request con 0x7ff7d803d5c0 auth_method 0 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adffff640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.6:6790/0 conn(0x7f4ae009eb80 legacy=0x7f4ae011ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adf7fe640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.6:6789/0 conn(0x7f4ae009ef60 legacy=0x7f4ae011f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753291754 0 0) 0x7f4ae00a0420 con 0x7f4ae009eb80 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): my global_id is 4271 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4acc003530 con 0x7f4ae009eb80 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419703794 0 0) 0x7f4ae0121930 con 0x7f4ae009ef60 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): my global_id is 4290 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae00a0420 con 0x7f4ae009ef60 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae48d5640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.203:6789/0 conn(0x7f4ae01396a0 legacy=0x7f4ae01d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3258756197 0 0) 0x7f4acc003530 con 0x7f4ae009eb80 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4ae0121930 con 0x7f4ae009eb80 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ad40037c0 con 0x7f4ae009eb80 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): dump: 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.430 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 244363736 0 0) 0x7f4ae00a0420 con 0x7f4ae009ef60 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4acc003530 con 0x7f4ae009ef60 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ad00019f0 con 0x7f4ae009ef60 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): dump: 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.431 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010502643 0 0) 0x7f4ae011fa10 con 0x7f4ae01396a0 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient(hunting): my global_id is 4246 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae00a0420 con 0x7f4ae01396a0 2024-04-25T21:40:08.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 294484831 0 0) 0x7f4ae0121930 con 0x7f4ae009eb80 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.203:6789/0 conn(0x7f4ae01396a0 legacy=0x7f4ae01d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.6:6789/0 conn(0x7f4ae009ef60 legacy=0x7f4ae011f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 monclient: found mon.c 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae011c410 con 0x7f4ae009eb80 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 5 monclient: authenticate success, global_id 4271 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 10 monclient: _renew_subs 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae0121090 con 0x7f4ae009eb80 2024-04-25T21:40:08.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 10 monclient: _renew_subs 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4ae011b900 con 0x7f4ae009eb80 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4ae0062e50 con 0x7f4ae009eb80 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ad4001c70 con 0x7f4ae009eb80 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ad4003f90 con 0x7f4ae009eb80 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 10 monclient: dump: 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f4ad40041b0 con 0x7f4ae009eb80 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.344+0000 7f4ade7fc640 4 set_mon_vals no callback set 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.348+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f4ad4043620 con 0x7f4ae009eb80 2024-04-25T21:40:08.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.348+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== osd.4 v1:172.21.15.203:6801/1405305648 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8a14041490 con 0x7f8a1403d820 2024-04-25T21:40:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.348+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6806/2126016572 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4acc0414c0 con 0x7f4acc03d850 2024-04-25T21:40:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.348+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f4ad4043860 con 0x7f4ae009eb80 2024-04-25T21:40:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.348+0000 7f4adcff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-04-25T21:40:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.348+0000 7f4adf7fe640 10 monclient: get_auth_request con 0x7f4acc036c70 auth_method 0 2024-04-25T21:40:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.348+0000 7f4ae48d5640 10 monclient: get_auth_request con 0x7f4acc03d850 auth_method 0 2024-04-25T21:40:08.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.352+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== osd.3 v1:172.21.15.203:6800/3201437079 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff7d8041190 con 0x7ff7d803d5c0 2024-04-25T21:40:08.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.352+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== osd.1 v1:172.21.15.6:6806/2126016572 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4acc0414c0 con 0x7f4acc03d850 2024-04-25T21:40:08.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 --> v1:172.21.15.203:6801/1405305648 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8a08002c60 con 0x7f8a1403d820 2024-04-25T21:40:08.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a0f7fe640 1 -- 172.21.15.6:0/1973264231 <== osd.4 v1:172.21.15.203:6801/1405305648 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 839772338) 0x7f8a08002c60 con 0x7f8a1403d820 2024-04-25T21:40:08.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.203:6801/1405305648 conn(0x7f8a1403d820 legacy=0x7f8a1403fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.203:6812/3164820986 conn(0x7f8a14036bd0 legacy=0x7f8a14039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a2d18c640 10 monclient: shutdown 2024-04-25T21:40:08.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a2d18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 >> v1:172.21.15.6:6790/0 conn(0x7f8a2813d0a0 legacy=0x7f8a281d83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.360+0000 7f8a2759e640 1 -- 172.21.15.6:0/1973264231 reap_dead start 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 shutdown_connections 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 >> 172.21.15.6:0/1973264231 conn(0x7f8a280a5f80 msgr2=0x7f8a281364b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 shutdown_connections 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f8a2d18c640 1 -- 172.21.15.6:0/1973264231 wait complete. 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 --> v1:172.21.15.203:6808/1687281622 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ed4002b20 con 0x7f0ed40c5cf0 2024-04-25T21:40:08.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0ec37fe640 1 -- 172.21.15.6:0/304069260 <== osd.5 v1:172.21.15.203:6808/1687281622 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7f0ed4002b20 con 0x7f0ed40c5cf0 2024-04-25T21:40:08.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 >> v1:172.21.15.203:6808/1687281622 conn(0x7f0ed40c5cf0 legacy=0x7f0ed4029730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 >> v1:172.21.15.203:6812/3164820986 conn(0x7f0ec8036b60 legacy=0x7f0ec8039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 10 monclient: shutdown 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 >> v1:172.21.15.6:6790/0 conn(0x7f0ed40b54d0 legacy=0x7f0ed40c4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edb7fe640 1 -- 172.21.15.6:0/304069260 reap_dead start 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 shutdown_connections 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 >> 172.21.15.6:0/304069260 conn(0x7f0ed402aaf0 msgr2=0x7f0ed4029020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 shutdown_connections 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.364+0000 7f0edbfff640 1 -- 172.21.15.6:0/304069260 wait complete. 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 --> v1:172.21.15.6:6806/2126016572 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4ae011bbf0 con 0x7f4acc03d850 2024-04-25T21:40:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4adcff9640 1 -- 172.21.15.6:0/1249852406 <== osd.1 v1:172.21.15.6:6806/2126016572 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 839772338) 0x7f4ae011bbf0 con 0x7f4acc03d850 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.6:6806/2126016572 conn(0x7f4acc03d850 legacy=0x7f4acc03fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.203:6812/3164820986 conn(0x7f4acc036c70 legacy=0x7f4acc039130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 10 monclient: shutdown 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 >> v1:172.21.15.6:6790/0 conn(0x7f4ae009eb80 legacy=0x7f4ae011ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae48d5640 1 -- 172.21.15.6:0/1249852406 reap_dead start 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 shutdown_connections 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 >> 172.21.15.6:0/1249852406 conn(0x7f4ae00a5710 msgr2=0x7f4ae0138a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 shutdown_connections 2024-04-25T21:40:08.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.368+0000 7f4ae50d6640 1 -- 172.21.15.6:0/1249852406 wait complete. 2024-04-25T21:40:08.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.376+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 --> v1:172.21.15.203:6800/3201437079 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff7ec1c8050 con 0x7ff7d803d5c0 2024-04-25T21:40:08.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7d3fff640 1 -- 172.21.15.6:0/1952468897 <== osd.3 v1:172.21.15.203:6800/3201437079 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7ff7ec1c8050 con 0x7ff7d803d5c0 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.203:6800/3201437079 conn(0x7ff7d803d5c0 legacy=0x7ff7d803f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.203:6812/3164820986 conn(0x7ff7d8036a10 legacy=0x7ff7d8038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 10 monclient: shutdown 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 >> v1:172.21.15.6:6790/0 conn(0x7ff7ec09d2a0 legacy=0x7ff7ec1d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7ebfff640 1 -- 172.21.15.6:0/1952468897 reap_dead start 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 shutdown_connections 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 >> 172.21.15.6:0/1952468897 conn(0x7ff7ec0a2440 msgr2=0x7ff7ec1344c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 shutdown_connections 2024-04-25T21:40:08.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.380+0000 7ff7f1ae7640 1 -- 172.21.15.6:0/1952468897 wait complete. 2024-04-25T21:40:08.446 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-04-25T21:40:08.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T21:40:08.447 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-04-25T21:40:08.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T21:40:08.448 INFO:teuthology.orchestra.run.smithi006.stdout:51539607556 2024-04-25T21:40:08.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T21:40:08.449 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2024-04-25T21:40:08.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient: monmap: 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.583783+0000 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.583783+0000 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.582 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.583 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.583 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient: monmap: 2024-04-25T21:40:08.583 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.584204+0000 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.584204+0000 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 1 Processor -- start 2024-04-25T21:40:08.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 1 -- start start 2024-04-25T21:40:08.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient: init 2024-04-25T21:40:08.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 1 Processor -- start 2024-04-25T21:40:08.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 1 -- start start 2024-04-25T21:40:08.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient: init 2024-04-25T21:40:08.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6ec06e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6ec09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f0c44640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa6ec06e890 0x7fa6ec0a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45636/0 (socket says 172.21.15.6:45636) 2024-04-25T21:40:08.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f1445640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6ec13d0a0 0x7fa6ec13f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47586/0 (socket says 172.21.15.6:47586) 2024-04-25T21:40:08.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6ec13d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f0c44640 1 -- 172.21.15.6:0/1231752472 learned_addr learned my addr 172.21.15.6:0/1231752472 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6ec09b0d0 con 0x7fa6ec13d0a0 2024-04-25T21:40:08.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6ec0a4ca0 con 0x7fa6ec09bab0 2024-04-25T21:40:08.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient(hunting): picked mon.noname-b con 0x7f9c3806e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient(hunting): picked mon.noname-c con 0x7f9c3809bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3813d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6ec06f070 con 0x7fa6ec06e890 2024-04-25T21:40:08.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3d53b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9c3806e890 0x7f9c380a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45638/0 (socket says 172.21.15.6:45638) 2024-04-25T21:40:08.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3d53b640 1 -- 172.21.15.6:0/2088300041 learned_addr learned my addr 172.21.15.6:0/2088300041 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3809b0d0 con 0x7f9c3813d0a0 2024-04-25T21:40:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c380a4ca0 con 0x7f9c3809bab0 2024-04-25T21:40:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3806f070 con 0x7f9c3806e890 2024-04-25T21:40:08.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7f9c3f79f640 10 monclient(hunting): authenticate will time out at 967.494572s 2024-04-25T21:40:08.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.580+0000 7fa6f2ea8640 10 monclient(hunting): authenticate will time out at 967.494617s 2024-04-25T21:40:08.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f1445640 1 -- 172.21.15.6:0/1231752472 >> v1:172.21.15.6:6789/0 conn(0x7fa6ec13d0a0 legacy=0x7fa6ec13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f0c44640 1 -- 172.21.15.6:0/1231752472 >> v1:172.21.15.6:6790/0 conn(0x7fa6ec06e890 legacy=0x7fa6ec0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6ebfff640 1 -- 172.21.15.6:0/1231752472 >> v1:172.21.15.203:6789/0 conn(0x7fa6ec09bab0 legacy=0x7fa6ec0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2649972486 0 0) 0x7fa6e40017e0 con 0x7fa6ec13d0a0 2024-04-25T21:40:08.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): my global_id is 4293 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6dc003530 con 0x7fa6ec13d0a0 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471341186 0 0) 0x7fa6ec06f070 con 0x7fa6ec06e890 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): my global_id is 4274 2024-04-25T21:40:08.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6e40017e0 con 0x7fa6ec06e890 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3d53b640 1 -- 172.21.15.6:0/2088300041 >> v1:172.21.15.6:6790/0 conn(0x7f9c3806e890 legacy=0x7f9c380a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3dd3c640 1 -- 172.21.15.6:0/2088300041 >> v1:172.21.15.6:6789/0 conn(0x7f9c3813d0a0 legacy=0x7f9c3813f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705243560 0 0) 0x7f9c3806f070 con 0x7f9c3806e890 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): my global_id is 4277 2024-04-25T21:40:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c24003530 con 0x7f9c3806e890 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2798914872 0 0) 0x7fa6ec0a4ca0 con 0x7fa6ec09bab0 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient(hunting): my global_id is 4249 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6ec06f070 con 0x7fa6ec09bab0 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2965051768 0 0) 0x7fa6e40017e0 con 0x7fa6ec06e890 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 >> v1:172.21.15.203:6789/0 conn(0x7fa6ec09bab0 legacy=0x7fa6ec0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 >> v1:172.21.15.6:6789/0 conn(0x7fa6ec13d0a0 legacy=0x7fa6ec13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 monclient: found mon.noname-b 2024-04-25T21:40:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec09adf0 con 0x7fa6ec06e890 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa6e0001800 con 0x7fa6ec06e890 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: dump: 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.595 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 5 monclient: authenticate success, global_id 4274 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395084048 0 0) 0x7f9c30001830 con 0x7f9c3813d0a0 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient(hunting): my global_id is 4296 2024-04-25T21:40:08.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c3806f070 con 0x7f9c3813d0a0 2024-04-25T21:40:08.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6e0001b50 con 0x7fa6ec06e890 2024-04-25T21:40:08.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 1 -- 172.21.15.6:0/1231752472 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa6e00048b0 con 0x7fa6ec06e890 2024-04-25T21:40:08.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6eb7fe640 10 monclient: dump: 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 4 set_mon_vals no callback set 2024-04-25T21:40:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2613955170 0 0) 0x7f9c24003530 con 0x7f9c3806e890 2024-04-25T21:40:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 >> v1:172.21.15.203:6789/0 conn(0x7f9c3809bab0 legacy=0x7f9c380a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 >> v1:172.21.15.6:6789/0 conn(0x7f9c3813d0a0 legacy=0x7f9c3813f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 monclient: found mon.noname-b 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3809adf0 con 0x7f9c3806e890 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9c340018d0 con 0x7f9c3806e890 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: dump: 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 5 monclient: authenticate success, global_id 4277 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c34001c20 con 0x7f9c3806e890 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 1 -- 172.21.15.6:0/2088300041 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9c34004a50 con 0x7f9c3806e890 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c2ffff640 10 monclient: dump: 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 4 set_mon_vals no callback set 2024-04-25T21:40:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 10 monclient: shutdown 2024-04-25T21:40:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 >> v1:172.21.15.6:6790/0 conn(0x7fa6ec06e890 legacy=0x7fa6ec0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 shutdown_connections 2024-04-25T21:40:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 >> 172.21.15.6:0/1231752472 conn(0x7fa6ec0a5f80 msgr2=0x7fa6ec0a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 shutdown_connections 2024-04-25T21:40:08.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/1231752472 wait complete. 2024-04-25T21:40:08.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 10 monclient: shutdown 2024-04-25T21:40:08.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 >> v1:172.21.15.6:6790/0 conn(0x7f9c3806e890 legacy=0x7f9c380a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 shutdown_connections 2024-04-25T21:40:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 >> 172.21.15.6:0/2088300041 conn(0x7f9c380a5f80 msgr2=0x7f9c380a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 shutdown_connections 2024-04-25T21:40:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 10 monclient: monmap: 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 1 Processor -- start 2024-04-25T21:40:08.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 1 -- start start 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 1 -- 172.21.15.6:0/2088300041 wait complete. 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 10 monclient: monmap: 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.607 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7f9c3f79f640 1 Processor -- start 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.584+0000 7fa6f2ea8640 10 monclient: init 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 1 -- start start 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient: init 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient(hunting): picked mon.noname-c con 0x7f9c3806e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient(hunting): picked mon.noname-b con 0x7f9c3809bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3813d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3809c490 con 0x7f9c3813d0a0 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c38071050 con 0x7f9c3809bab0 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c38072aa0 con 0x7f9c3806e890 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3f79f640 10 monclient(hunting): authenticate will time out at 967.500485s 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3d53b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f9c3806e890 0x7f9c3813be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45644/0 (socket says 172.21.15.6:45644) 2024-04-25T21:40:08.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3d53b640 1 -- 172.21.15.6:0/4121149 learned_addr learned my addr 172.21.15.6:0/4121149 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6ec06e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f0c44640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6ec06e890 0x7fa6ec1cbef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47616/0 (socket says 172.21.15.6:47616) 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6ec09bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6ebfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa6ec09bab0 0x7fa6ec1d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45654/0 (socket says 172.21.15.6:45654) 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6ebfff640 1 -- 172.21.15.6:0/2439560355 learned_addr learned my addr 172.21.15.6:0/2439560355 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6ec13d0a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6ec13f910 con 0x7fa6ec06e890 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6ec09c490 con 0x7fa6ec13d0a0 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6ec0a1110 con 0x7fa6ec09bab0 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f2ea8640 10 monclient(hunting): authenticate will time out at 967.501326s 2024-04-25T21:40:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3d53b640 1 -- 172.21.15.6:0/4121149 >> v1:172.21.15.6:6790/0 conn(0x7f9c3806e890 legacy=0x7f9c3813be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3dd3c640 1 -- 172.21.15.6:0/4121149 >> v1:172.21.15.6:6789/0 conn(0x7f9c3813d0a0 legacy=0x7f9c381d8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 839178131 0 0) 0x7f9c38072aa0 con 0x7f9c3806e890 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): my global_id is 4280 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c1c003530 con 0x7f9c3806e890 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c3cd3a640 1 -- 172.21.15.6:0/4121149 >> v1:172.21.15.203:6789/0 conn(0x7f9c3809bab0 legacy=0x7f9c381d4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3283829153 0 0) 0x7f9c3809c490 con 0x7f9c3813d0a0 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): my global_id is 4299 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c38072aa0 con 0x7f9c3813d0a0 2024-04-25T21:40:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 777084631 0 0) 0x7f9c1c003530 con 0x7f9c3806e890 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c3809c490 con 0x7f9c3806e890 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9c34004990 con 0x7f9c3806e890 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): dump: 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.614 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7f9c2dffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485484574 0 0) 0x7f9c38071050 con 0x7f9c3809bab0 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient(hunting): my global_id is 4252 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c1c003530 con 0x7f9c3809bab0 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2346389659 0 0) 0x7f9c3809c490 con 0x7f9c3806e890 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 >> v1:172.21.15.203:6789/0 conn(0x7f9c3809bab0 legacy=0x7f9c381d4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 >> v1:172.21.15.6:6789/0 conn(0x7f9c3813d0a0 legacy=0x7f9c381d8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 monclient: found mon.c 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c38072d80 con 0x7f9c3806e890 2024-04-25T21:40:08.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6f0c44640 1 -- 172.21.15.6:0/2439560355 >> v1:172.21.15.6:6789/0 conn(0x7fa6ec06e890 legacy=0x7fa6ec1cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108971255 0 0) 0x7fa6ec13f910 con 0x7fa6ec06e890 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): my global_id is 4302 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6d0003530 con 0x7fa6ec06e890 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6ebfff640 1 -- 172.21.15.6:0/2439560355 >> v1:172.21.15.6:6790/0 conn(0x7fa6ec09bab0 legacy=0x7fa6ec1d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1446346525 0 0) 0x7fa6d0003530 con 0x7fa6ec06e890 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa6ec13f910 con 0x7fa6ec06e890 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa6e00047f0 con 0x7fa6ec06e890 2024-04-25T21:40:08.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): dump: 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.588+0000 7fa6e97fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4233026281 0 0) 0x7fa6ec13f910 con 0x7fa6ec06e890 2024-04-25T21:40:08.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 >> v1:172.21.15.6:6790/0 conn(0x7fa6ec09bab0 legacy=0x7fa6ec1d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 >> v1:172.21.15.203:6789/0 conn(0x7fa6ec13d0a0 legacy=0x7fa6ec1d4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 monclient: found mon.a 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec0a13f0 con 0x7fa6ec06e890 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa6e0004e60 con 0x7fa6ec06e890 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa6e00051b0 con 0x7fa6ec06e890 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 10 monclient: dump: 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6eaffd640 4 set_mon_vals no callback set 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 5 monclient: authenticate success, global_id 4302 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 10 monclient: _renew_subs 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ec140270 con 0x7fa6ec06e890 2024-04-25T21:40:08.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 10 monclient: _renew_subs 2024-04-25T21:40:08.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 5 monclient: authenticate success, global_id 4280 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 10 monclient: _renew_subs 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c381d0a70 con 0x7f9c3806e890 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 10 monclient: _renew_subs 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c38072080 con 0x7f9c3806e890 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c34001c20 con 0x7f9c3806e890 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c38062e50 con 0x7f9c3806e890 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9c340050e0 con 0x7f9c3806e890 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2f7fe640 4 set_mon_vals no callback set 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 10 monclient: dump: 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.623 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.624 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa6ec13e660 con 0x7fa6ec06e890 2024-04-25T21:40:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f9c3400fe80 con 0x7f9c3806e890 2024-04-25T21:40:08.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f9c34044350 con 0x7f9c3806e890 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fa6e000ff50 con 0x7fa6ec06e890 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6ec0a0f10 con 0x7fa6ec06e890 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.592+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7fa6e0044280 con 0x7fa6ec06e890 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.596+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f9c340157b0 con 0x7f9c3806e890 2024-04-25T21:40:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.596+0000 7f9c2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.596+0000 7f9c2dffb640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.600+0000 7f9c3cd3a640 10 monclient: get_auth_request con 0x7f9c1c036920 auth_method 0 2024-04-25T21:40:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.600+0000 7fa6ebfff640 10 monclient: get_auth_request con 0x7fa6d0036b00 auth_method 0 2024-04-25T21:40:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.604+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fa6e0015660 con 0x7fa6ec06e890 2024-04-25T21:40:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.604+0000 7fa6e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.604+0000 7fa6e97fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:08.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient: monmap: 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.632768+0000 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.632768+0000 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.631 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 1 Processor -- start 2024-04-25T21:40:08.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 1 -- start start 2024-04-25T21:40:08.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient: init 2024-04-25T21:40:08.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient(hunting): picked mon.noname-b con 0x7fafa406e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient(hunting): picked mon.noname-c con 0x7fafa409bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafa8c9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fafa406e890 0x7fafa40a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45668/0 (socket says 172.21.15.6:45668) 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient: monmap: 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.634166+0000 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.634166+0000 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.634 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fe4b6cdf640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fe4b6cdf640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fe4b6cdf640 10 monclient: monmap: 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.634502+0000 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.634502+0000 2024-04-25T21:40:08.635 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.636 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafa3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fafa409bab0 0x7fafa40a4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.6:54418/0 (socket says 172.21.15.6:54418) 2024-04-25T21:40:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafa949c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fafa413d0a0 0x7fafa413f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47618/0 (socket says 172.21.15.6:47618) 2024-04-25T21:40:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient(hunting): picked mon.noname-a con 0x7fafa413d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafa8c9b640 1 -- 172.21.15.6:0/1767957883 learned_addr learned my addr 172.21.15.6:0/1767957883 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa409b0d0 con 0x7fafa413d0a0 2024-04-25T21:40:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 1 Processor -- start 2024-04-25T21:40:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa40a4ca0 con 0x7fafa409bab0 2024-04-25T21:40:08.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa406f070 con 0x7fafa406e890 2024-04-25T21:40:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fafaaeff640 10 monclient(hunting): authenticate will time out at 967.543328s 2024-04-25T21:40:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 1 -- start start 2024-04-25T21:40:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient: init 2024-04-25T21:40:08.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fe4b6cdf640 1 Processor -- start 2024-04-25T21:40:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fe4b6cdf640 1 -- start start 2024-04-25T21:40:08.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7fe4b6cdf640 10 monclient: init 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient(hunting): picked mon.noname-a con 0x7f15b006e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient(hunting): picked mon.noname-c con 0x7f15b009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 10 monclient(hunting): picked mon.noname-b con 0x7f15b013d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.628+0000 7f15b61d0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b009b0d0 con 0x7f15b006e890 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b00a4ca0 con 0x7f15b009bab0 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b006f070 con 0x7f15b013d0a0 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 10 monclient(hunting): authenticate will time out at 967.544066s 2024-04-25T21:40:08.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15affff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f15b013d0a0 0x7f15b013f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45672/0 (socket says 172.21.15.6:45672) 2024-04-25T21:40:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15affff640 1 -- 172.21.15.6:0/3899763532 learned_addr learned my addr 172.21.15.6:0/3899763532 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b006e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b009bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b013d0a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b009b0d0 con 0x7fe4b006e890 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b4a7b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe4b006e890 0x7fe4b00a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47630/0 (socket says 172.21.15.6:47630) 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b4a7b640 1 -- 172.21.15.6:0/2566719331 learned_addr learned my addr 172.21.15.6:0/2566719331 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/2566719331 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b00a4ca0 con 0x7fe4b013d0a0 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/2566719331 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b006f070 con 0x7fe4b009bab0 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient(hunting): authenticate will time out at 967.544723s 2024-04-25T21:40:08.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa8c9b640 1 -- 172.21.15.6:0/1767957883 >> v1:172.21.15.6:6790/0 conn(0x7fafa406e890 legacy=0x7fafa40a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa949c640 1 -- 172.21.15.6:0/1767957883 >> v1:172.21.15.6:6789/0 conn(0x7fafa413d0a0 legacy=0x7fafa413f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1857975235 0 0) 0x7fafa406f070 con 0x7fafa406e890 2024-04-25T21:40:08.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa3fff640 1 -- 172.21.15.6:0/1767957883 >> v1:172.21.15.203:6789/0 conn(0x7fafa409bab0 legacy=0x7fafa40a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient(hunting): my global_id is 4286 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf94003530 con 0x7fafa406e890 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3364836889 0 0) 0x7faf94003530 con 0x7fafa406e890 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 >> v1:172.21.15.203:6789/0 conn(0x7fafa409bab0 legacy=0x7fafa40a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 >> v1:172.21.15.6:6789/0 conn(0x7fafa413d0a0 legacy=0x7fafa413f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 monclient: found mon.noname-b 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa409adf0 con 0x7fafa406e890 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7faf980018d0 con 0x7fafa406e890 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: dump: 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 5 monclient: authenticate success, global_id 4286 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fc55c772640 10 monclient: get_monmap_and_config 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fc55c772640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fc55c772640 10 monclient: monmap: 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:08.637260+0000 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:08.637260+0000 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.646 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faf98001c20 con 0x7fafa406e890 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 1 -- 172.21.15.6:0/1767957883 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7faf98004a50 con 0x7fafa406e890 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafa37fe640 10 monclient: dump: 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 4 set_mon_vals no callback set 2024-04-25T21:40:08.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15affff640 1 -- 172.21.15.6:0/3899763532 >> v1:172.21.15.6:6790/0 conn(0x7f15b013d0a0 legacy=0x7f15b013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814945516 0 0) 0x7f15b006f070 con 0x7f15b013d0a0 2024-04-25T21:40:08.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient(hunting): my global_id is 4289 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1598003530 con 0x7f15b013d0a0 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1791466478 0 0) 0x7f1598003530 con 0x7f15b013d0a0 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 >> v1:172.21.15.203:6789/0 conn(0x7f15b009bab0 legacy=0x7f15b00a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 >> v1:172.21.15.6:6789/0 conn(0x7f15b006e890 legacy=0x7f15b00a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 monclient: found mon.noname-b 2024-04-25T21:40:08.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b009adf0 con 0x7f15b013d0a0 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 10 monclient: shutdown 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 >> v1:172.21.15.6:6790/0 conn(0x7fafa406e890 legacy=0x7fafa40a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15a00018d0 con 0x7f15b013d0a0 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: dump: 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.652 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 5 monclient: authenticate success, global_id 4289 2024-04-25T21:40:08.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 shutdown_connections 2024-04-25T21:40:08.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 >> 172.21.15.6:0/1767957883 conn(0x7fafa40a5f80 msgr2=0x7fafa40a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b4a7b640 1 -- 172.21.15.6:0/2566719331 >> v1:172.21.15.6:6789/0 conn(0x7fe4b006e890 legacy=0x7fe4b00a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15a0001c20 con 0x7f15b013d0a0 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 1 -- 172.21.15.6:0/3899763532 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15a0004a50 con 0x7f15b013d0a0 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15ae7fc640 10 monclient: dump: 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.654 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 4 set_mon_vals no callback set 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 shutdown_connections 2024-04-25T21:40:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 1 -- 172.21.15.6:0/1767957883 wait complete. 2024-04-25T21:40:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 21879789 0 0) 0x7fe4a4001460 con 0x7fe4b006e890 2024-04-25T21:40:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient(hunting): my global_id is 4308 2024-04-25T21:40:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4a0003530 con 0x7fe4b006e890 2024-04-25T21:40:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 21754238 0 0) 0x7fe4a0003530 con 0x7fe4b006e890 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 >> v1:172.21.15.6:6790/0 conn(0x7fe4b009bab0 legacy=0x7fe4b00a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 >> v1:172.21.15.203:6789/0 conn(0x7fe4b013d0a0 legacy=0x7fe4b013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 monclient: found mon.noname-a 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b009adf0 con 0x7fe4b006e890 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe4a4001aa0 con 0x7fe4b006e890 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: dump: 2024-04-25T21:40:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 5 monclient: authenticate success, global_id 4308 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4a40048b0 con 0x7fe4b006e890 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 1 -- 172.21.15.6:0/2566719331 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe4a4004c40 con 0x7fe4b006e890 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4b6cdf640 4 set_mon_vals no callback set 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fe4af7fe640 10 monclient: dump: 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.659 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.660 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 10 monclient: shutdown 2024-04-25T21:40:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 1 -- 172.21.15.6:0/3899763532 >> v1:172.21.15.6:6790/0 conn(0x7f15b013d0a0 legacy=0x7f15b013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 1 -- 172.21.15.6:0/3899763532 shutdown_connections 2024-04-25T21:40:08.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7f15b61d0640 1 -- 172.21.15.6:0/3899763532 >> 172.21.15.6:0/3899763532 conn(0x7f15b00a5f80 msgr2=0x7f15b00a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.632+0000 7fafaaeff640 10 monclient: monmap: 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.661 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 1 -- 172.21.15.6:0/3899763532 shutdown_connections 2024-04-25T21:40:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 1 -- 172.21.15.6:0/3899763532 wait complete. 2024-04-25T21:40:08.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 1 Processor -- start 2024-04-25T21:40:08.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient: shutdown 2024-04-25T21:40:08.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/2566719331 >> v1:172.21.15.6:6789/0 conn(0x7fe4b006e890 legacy=0x7fe4b00a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 1 -- start start 2024-04-25T21:40:08.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient: init 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/2566719331 shutdown_connections 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/2566719331 >> 172.21.15.6:0/2566719331 conn(0x7fe4b00a5f80 msgr2=0x7fe4b00a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/2566719331 shutdown_connections 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/2566719331 wait complete. 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient: monmap: 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.664 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.665 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient(hunting): picked mon.noname-c con 0x7fafa406e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient(hunting): picked mon.noname-b con 0x7fafa409bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient(hunting): picked mon.noname-a con 0x7fafa413d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa413c5c0 con 0x7fafa413d0a0 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa40a9190 con 0x7fafa409bab0 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa4072200 con 0x7fafa406e890 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa8c9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fafa406e890 0x7fafa413beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45694/0 (socket says 172.21.15.6:45694) 2024-04-25T21:40:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa8c9b640 1 -- 172.21.15.6:0/3220807399 learned_addr learned my addr 172.21.15.6:0/3220807399 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 1 Processor -- start 2024-04-25T21:40:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafaaeff640 10 monclient(hunting): authenticate will time out at 967.549158s 2024-04-25T21:40:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 1 -- start start 2024-04-25T21:40:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient: init 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient: monmap: 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.669 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 Processor -- start 2024-04-25T21:40:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- start start 2024-04-25T21:40:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient: init 2024-04-25T21:40:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient(hunting): picked mon.noname-a con 0x7f15b006e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient(hunting): picked mon.noname-c con 0x7f15b009bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient(hunting): picked mon.noname-b con 0x7f15b013d0a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b0138750 con 0x7f15b006e890 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15aeffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f15b009bab0 0x7f15b00a1220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45706/0 (socket says 172.21.15.6:45706) 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15aeffd640 1 -- 172.21.15.6:0/2667659341 learned_addr learned my addr 172.21.15.6:0/2667659341 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b0138330 con 0x7f15b013d0a0 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b00a1930 con 0x7f15b009bab0 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7f15b61d0640 10 monclient(hunting): authenticate will time out at 967.550040s 2024-04-25T21:40:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b006e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b4a7b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe4b006e890 0x7fe4b01cbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47666/0 (socket says 172.21.15.6:47666) 2024-04-25T21:40:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b013d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b4a7b640 1 -- 172.21.15.6:0/4279242839 learned_addr learned my addr 172.21.15.6:0/4279242839 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b009c490 con 0x7fe4b006e890 2024-04-25T21:40:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b00a91d0 con 0x7fe4b009bab0 2024-04-25T21:40:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b0123f30 con 0x7fe4b013d0a0 2024-04-25T21:40:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fe4b6cdf640 10 monclient(hunting): authenticate will time out at 967.550473s 2024-04-25T21:40:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa8c9b640 1 -- 172.21.15.6:0/3220807399 >> v1:172.21.15.6:6790/0 conn(0x7fafa406e890 legacy=0x7fafa413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa949c640 1 -- 172.21.15.6:0/3220807399 >> v1:172.21.15.6:6789/0 conn(0x7fafa413d0a0 legacy=0x7fafa41d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2269305224 0 0) 0x7fafa4072200 con 0x7fafa406e890 2024-04-25T21:40:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): my global_id is 4292 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf88003530 con 0x7fafa406e890 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa3fff640 1 -- 172.21.15.6:0/3220807399 >> v1:172.21.15.203:6789/0 conn(0x7fafa409bab0 legacy=0x7fafa41d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1834121889 0 0) 0x7faf88003530 con 0x7fafa406e890 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafa4072200 con 0x7fafa406e890 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7faf98004990 con 0x7fafa406e890 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): dump: 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.636+0000 7fafa17fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 189712435 0 0) 0x7fafa4072200 con 0x7fafa406e890 2024-04-25T21:40:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 1 Processor -- start 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 >> v1:172.21.15.203:6789/0 conn(0x7fafa409bab0 legacy=0x7fafa41d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 >> v1:172.21.15.6:6789/0 conn(0x7fafa413d0a0 legacy=0x7fafa41d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 monclient: found mon.c 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa409c490 con 0x7fafa406e890 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 5 monclient: authenticate success, global_id 4292 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 10 monclient: _renew_subs 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fafa4128580 con 0x7fafa406e890 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 10 monclient: _renew_subs 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fafa4071ef0 con 0x7fafa406e890 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 1 -- start start 2024-04-25T21:40:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient: init 2024-04-25T21:40:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15aeffd640 1 -- 172.21.15.6:0/2667659341 >> v1:172.21.15.6:6790/0 conn(0x7f15b009bab0 legacy=0x7f15b00a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faf98001c00 con 0x7fafa406e890 2024-04-25T21:40:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7faf98004f80 con 0x7fafa406e890 2024-04-25T21:40:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 10 monclient: dump: 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa2ffd640 4 set_mon_vals no callback set 2024-04-25T21:40:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7faf980051a0 con 0x7fafa406e890 2024-04-25T21:40:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient(hunting): picked mon.noname-b con 0x7fc554098490 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient(hunting): picked mon.noname-a con 0x7fc554098870 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient(hunting): picked mon.noname-c con 0x7fc554098c90 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55409a780 con 0x7fc554098870 2024-04-25T21:40:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55409aa60 con 0x7fc554098c90 2024-04-25T21:40:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55409ad40 con 0x7fc554098490 2024-04-25T21:40:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4b4a7b640 1 -- 172.21.15.6:0/4279242839 >> v1:172.21.15.6:6789/0 conn(0x7fe4b006e890 legacy=0x7fe4b01cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15af7fe640 1 -- 172.21.15.6:0/2667659341 >> v1:172.21.15.6:6789/0 conn(0x7f15b006e890 legacy=0x7f15b00a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7faf98044440 con 0x7fafa406e890 2024-04-25T21:40:08.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55a50e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fc554098490 0x7fc55409ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45722/0 (socket says 172.21.15.6:45722) 2024-04-25T21:40:08.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55a50e640 1 -- 172.21.15.6:0/3482336118 learned_addr learned my addr 172.21.15.6:0/3482336118 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305913322 0 0) 0x7f15b00a1930 con 0x7f15b009bab0 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): my global_id is 4295 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f159c003530 con 0x7f15b009bab0 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf84005090 con 0x7fafa406e890 2024-04-25T21:40:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fafa3fff640 10 monclient: get_auth_request con 0x7faf88036b20 auth_method 0 2024-04-25T21:40:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20225728 0 0) 0x7f15a8003ed0 con 0x7f15b006e890 2024-04-25T21:40:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55c772640 10 monclient(hunting): authenticate will time out at 967.553908s 2024-04-25T21:40:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): my global_id is 4314 2024-04-25T21:40:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15b00a1930 con 0x7f15b006e890 2024-04-25T21:40:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2150362325 0 0) 0x7fe4b009c490 con 0x7fe4b006e890 2024-04-25T21:40:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): my global_id is 4317 2024-04-25T21:40:08.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe494003530 con 0x7fe4b006e890 2024-04-25T21:40:08.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4affff640 1 -- 172.21.15.6:0/4279242839 >> v1:172.21.15.203:6789/0 conn(0x7fe4b009bab0 legacy=0x7fe4b01d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15affff640 1 -- 172.21.15.6:0/2667659341 >> v1:172.21.15.203:6789/0 conn(0x7f15b013d0a0 legacy=0x7f15b01d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1488820514 0 0) 0x7fe494003530 con 0x7fe4b006e890 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe4b009c490 con 0x7fe4b006e890 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe4a40014d0 con 0x7fe4b006e890 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.689 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1350943798 0 0) 0x7f15b00a1930 con 0x7f15b006e890 2024-04-25T21:40:08.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15a8003ed0 con 0x7f15b006e890 2024-04-25T21:40:08.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15a8004750 con 0x7f15b006e890 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): dump: 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15affff640 1 --1- 172.21.15.6:0/2667659341 >> v1:172.21.15.203:6789/0 conn(0x7f15b013d0a0 0x7f15b01d3e80 :-1 s=OPENED pgs=103 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f15b0138330 auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2555280163 0 0) 0x7fe4b009c490 con 0x7fe4b006e890 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 >> v1:172.21.15.6:6790/0 conn(0x7fe4b013d0a0 legacy=0x7fe4b01d4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 >> v1:172.21.15.203:6789/0 conn(0x7fe4b009bab0 legacy=0x7fe4b01d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 monclient: found mon.a 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b0124210 con 0x7fe4b006e890 2024-04-25T21:40:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3181672473 0 0) 0x7f15a8003ed0 con 0x7f15b006e890 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 >> v1:172.21.15.6:6790/0 conn(0x7f15b009bab0 legacy=0x7f15b00a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 >> v1:172.21.15.203:6789/0 conn(0x7f15b013d0a0 legacy=0x7f15b01d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 monclient: found mon.a 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b00a8ef0 con 0x7f15b006e890 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 5 monclient: authenticate success, global_id 4314 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 10 monclient: _renew_subs 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15b00a2750 con 0x7f15b006e890 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742536348 0 0) 0x7f15b0138330 con 0x7f15b013d0a0 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f1597fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 10 monclient: _renew_subs 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f15b00a3030 con 0x7f15b006e890 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b0062e50 con 0x7f15b006e890 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe4a4004d90 con 0x7fe4b006e890 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe4a40050e0 con 0x7fe4b006e890 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4ad7fa640 10 monclient: dump: 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.697 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fe4aeffd640 4 set_mon_vals no callback set 2024-04-25T21:40:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.640+0000 7fc55a50e640 1 -- 172.21.15.6:0/3482336118 >> v1:172.21.15.6:6790/0 conn(0x7fc554098490 legacy=0x7fc55409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.644+0000 7fc55ad0f640 1 -- 172.21.15.6:0/3482336118 >> v1:172.21.15.203:6789/0 conn(0x7fc554098c90 legacy=0x7fc55413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.644+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7faf98015820 con 0x7fafa406e890 2024-04-25T21:40:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.644+0000 7fafa17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.644+0000 7fafa17fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 5 monclient: authenticate success, global_id 4317 2024-04-25T21:40:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 10 monclient: _renew_subs 2024-04-25T21:40:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b0122800 con 0x7fe4b006e890 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 10 monclient: _renew_subs 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe4b0121330 con 0x7fe4b006e890 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b0062e50 con 0x7fe4b006e890 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15a8001670 con 0x7f15b006e890 2024-04-25T21:40:08.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15a80019c0 con 0x7f15b006e890 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 10 monclient: dump: 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f15adffb640 4 set_mon_vals no callback set 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f15a8001b60 con 0x7f15b006e890 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f15a8044800 con 0x7f15b006e890 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3717836748 0 0) 0x7fc55409aa60 con 0x7fc554098c90 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): my global_id is 4267 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc540003530 con 0x7fc554098c90 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2207620699 0 0) 0x7fc55409ad40 con 0x7fc554098490 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient(hunting): my global_id is 4298 2024-04-25T21:40:08.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc55409aa60 con 0x7fc554098490 2024-04-25T21:40:08.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f15aeffd640 10 monclient: get_auth_request con 0x7f159c036bd0 auth_method 0 2024-04-25T21:40:08.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4084114051 0 0) 0x7fc540003530 con 0x7fc554098c90 2024-04-25T21:40:08.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 >> v1:172.21.15.6:6790/0 conn(0x7fc554098490 legacy=0x7fc55409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 >> v1:172.21.15.6:6789/0 conn(0x7fc554098870 legacy=0x7fc554137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 monclient: found mon.noname-c 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5540a1160 con 0x7fc554098c90 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc54c001bd0 con 0x7fc554098c90 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f15b0138750 con 0x7f15b006e890 2024-04-25T21:40:08.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7f1597fff640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55950c640 10 monclient: dump: 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55c772640 5 monclient: authenticate success, global_id 4267 2024-04-25T21:40:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fc55c772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:08.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc54c001f90 con 0x7fc554098c90 2024-04-25T21:40:08.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fc55950c640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fc55950c640 1 -- 172.21.15.6:0/3482336118 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc54c003b40 con 0x7fc554098c90 2024-04-25T21:40:08.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.648+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fe4a40053c0 con 0x7fe4b006e890 2024-04-25T21:40:08.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7fe4a4044310 con 0x7fe4b006e890 2024-04-25T21:40:08.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fc55950c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fc55950c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:08.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fe4affff640 10 monclient: get_auth_request con 0x7fe494036b00 auth_method 0 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.652+0000 7fc55950c640 10 monclient: dump: 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.709 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 4 set_mon_vals no callback set 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 10 monclient: shutdown 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 1 -- 172.21.15.6:0/3482336118 >> v1:172.21.15.203:6789/0 conn(0x7fc554098c90 legacy=0x7fc55413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fe4b009b0d0 con 0x7fe4b006e890 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fe4ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fe4ad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:08.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 1 -- 172.21.15.6:0/3482336118 shutdown_connections 2024-04-25T21:40:08.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 1 -- 172.21.15.6:0/3482336118 >> 172.21.15.6:0/3482336118 conn(0x7fc5540a1c50 msgr2=0x7fc5540a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 1 -- 172.21.15.6:0/3482336118 shutdown_connections 2024-04-25T21:40:08.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 1 -- 172.21.15.6:0/3482336118 wait complete. 2024-04-25T21:40:08.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 10 monclient: build_initial_monmap 2024-04-25T21:40:08.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 10 monclient: monmap: 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:08.715 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 1 Processor -- start 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 1 -- start start 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.656+0000 7fc55c772640 10 monclient: init 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 10 monclient(hunting): picked mon.noname-a con 0x7fc554098490 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 10 monclient(hunting): picked mon.noname-b con 0x7fc554098870 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 10 monclient(hunting): picked mon.noname-c con 0x7fc554098c90 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55412e060 con 0x7fc554098490 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55412e900 con 0x7fc554098870 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55a50e640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc554098490 0x7fc55413bcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47682/0 (socket says 172.21.15.6:47682) 2024-04-25T21:40:08.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55a50e640 1 -- 172.21.15.6:0/2884298377 learned_addr learned my addr 172.21.15.6:0/2884298377 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5541336d0 con 0x7fc554098c90 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55c772640 10 monclient(hunting): authenticate will time out at 967.573322s 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55a50e640 1 -- 172.21.15.6:0/2884298377 >> v1:172.21.15.6:6789/0 conn(0x7fc554098490 legacy=0x7fc55413bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc55ad0f640 1 -- 172.21.15.6:0/2884298377 >> v1:172.21.15.6:6790/0 conn(0x7fc554098c90 legacy=0x7fc5541cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1336969819 0 0) 0x7fc5541336d0 con 0x7fc554098c90 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): my global_id is 4301 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc538003530 con 0x7fc554098c90 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286698034 0 0) 0x7fc550001bc0 con 0x7fc554098490 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 10 monclient(hunting): my global_id is 4320 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.660+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5541336d0 con 0x7fc554098490 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc559d0d640 1 -- 172.21.15.6:0/2884298377 >> v1:172.21.15.203:6789/0 conn(0x7fc554098870 legacy=0x7fc55413c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1007396511 0 0) 0x7fc538003530 con 0x7fc554098c90 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc550001bc0 con 0x7fc554098c90 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc54c003b40 con 0x7fc554098c90 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient(hunting): dump: 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.718 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1965168338 0 0) 0x7fc5541336d0 con 0x7fc554098490 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc538003530 con 0x7fc554098490 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc550004990 con 0x7fc554098490 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient(hunting): dump: 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.719 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1351910931 0 0) 0x7fc550001bc0 con 0x7fc554098c90 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 >> v1:172.21.15.203:6789/0 conn(0x7fc554098870 legacy=0x7fc55413c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 >> v1:172.21.15.6:6789/0 conn(0x7fc554098490 legacy=0x7fc55413bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 monclient: found mon.c 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc554130220 con 0x7fc554098c90 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: _finish_auth 0 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 5 monclient: authenticate success, global_id 4301 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 10 monclient: _renew_subs 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc554132bb0 con 0x7fc554098c90 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 10 monclient: _renew_subs 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc55412d410 con 0x7fc554098c90 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc54c001f20 con 0x7fc554098c90 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc558d0b640 4 set_mon_vals no callback set 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fc54c0043a0 con 0x7fc554098c90 2024-04-25T21:40:08.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 10 monclient: dump: 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc53c005090 con 0x7fc554098c90 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fc54c004600 con 0x7fc554098c90 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7fc54c0434e0 con 0x7fc554098c90 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.664+0000 7fc55a50e640 10 monclient: get_auth_request con 0x7fc538036990 auth_method 0 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.676+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fc54c0159f0 con 0x7fc554098c90 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.676+0000 7fc54affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.676+0000 7fc54affd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-25T21:40:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-25T21:40:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe4b006b210 con 0x7fe4b006e890 2024-04-25T21:40:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4ad7fa640 1 -- 172.21.15.6:0/4279242839 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 727498649) 0x7fe4a4015690 con 0x7fe4b006e890 2024-04-25T21:40:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-25T21:40:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4ad7fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:08.823 INFO:teuthology.orchestra.run.smithi006.stdout:51539607554 2024-04-25T21:40:08.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 >> v1:172.21.15.203:6812/3164820986 conn(0x7fe494036b00 legacy=0x7fe494038fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 10 monclient: shutdown 2024-04-25T21:40:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 >> v1:172.21.15.6:6789/0 conn(0x7fe4b006e890 legacy=0x7fe4b01cbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 shutdown_connections 2024-04-25T21:40:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 >> 172.21.15.6:0/4279242839 conn(0x7fe4b00a5f80 msgr2=0x7fe4b013f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 shutdown_connections 2024-04-25T21:40:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.820+0000 7fe4b6cdf640 1 -- 172.21.15.6:0/4279242839 wait complete. 2024-04-25T21:40:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.832+0000 7fc55c772640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-25T21:40:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.832+0000 7fc55c772640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-25T21:40:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.832+0000 7fc55c772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.832+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc53c005380 con 0x7fc554098c90 2024-04-25T21:40:08.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc54affd640 1 -- 172.21.15.6:0/2884298377 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 727498649) 0x7fc54c0152f0 con 0x7fc554098c90 2024-04-25T21:40:08.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc54affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-25T21:40:08.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc54affd640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:08.838 INFO:teuthology.orchestra.run.smithi006.stdout:51539607554 2024-04-25T21:40:08.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607554 for osd.0 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 >> v1:172.21.15.203:6812/3164820986 conn(0x7fc538036990 legacy=0x7fc538038e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 10 monclient: shutdown 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 >> v1:172.21.15.6:6790/0 conn(0x7fc554098c90 legacy=0x7fc5541cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 shutdown_connections 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 >> 172.21.15.6:0/2884298377 conn(0x7fc5540a1c50 msgr2=0x7fc554133910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 shutdown_connections 2024-04-25T21:40:08.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.836+0000 7fc55c772640 1 -- 172.21.15.6:0/2884298377 wait complete. 2024-04-25T21:40:08.854 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607554 for osd.1 2024-04-25T21:40:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-25T21:40:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-25T21:40:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9c3806b210 con 0x7f9c3806e890 2024-04-25T21:40:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c2dffb640 1 -- 172.21.15.6:0/4121149 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 727498649) 0x7f9c34019400 con 0x7f9c3806e890 2024-04-25T21:40:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-25T21:40:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c2dffb640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:08.903 INFO:teuthology.orchestra.run.smithi006.stdout:51539607554 2024-04-25T21:40:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 >> v1:172.21.15.203:6812/3164820986 conn(0x7f9c1c036920 legacy=0x7f9c1c038de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 10 monclient: shutdown 2024-04-25T21:40:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 >> v1:172.21.15.6:6790/0 conn(0x7f9c3806e890 legacy=0x7f9c3813be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 shutdown_connections 2024-04-25T21:40:08.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 >> 172.21.15.6:0/4121149 conn(0x7f9c380a5f80 msgr2=0x7f9c3809fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 shutdown_connections 2024-04-25T21:40:08.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.900+0000 7f9c3f79f640 1 -- 172.21.15.6:0/4121149 wait complete. 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa6ec062e50 con 0x7fa6ec06e890 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6e97fa640 1 -- 172.21.15.6:0/2439560355 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 727498649) 0x7fa6e0044020 con 0x7fa6ec06e890 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6e97fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:08.907 INFO:teuthology.orchestra.run.smithi006.stdout:51539607554 2024-04-25T21:40:08.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 >> v1:172.21.15.203:6812/3164820986 conn(0x7fa6d0036b00 legacy=0x7fa6d0038fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 10 monclient: shutdown 2024-04-25T21:40:08.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 >> v1:172.21.15.6:6789/0 conn(0x7fa6ec06e890 legacy=0x7fa6ec1cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 shutdown_connections 2024-04-25T21:40:08.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 >> 172.21.15.6:0/2439560355 conn(0x7fa6ec0a5f80 msgr2=0x7fa6ec0a8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.904+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 shutdown_connections 2024-04-25T21:40:08.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.908+0000 7fa6f2ea8640 1 -- 172.21.15.6:0/2439560355 wait complete. 2024-04-25T21:40:08.922 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607554 for osd.2 2024-04-25T21:40:08.926 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607554 for osd.4 2024-04-25T21:40:08.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.964+0000 7fafaaeff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-25T21:40:08.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.964+0000 7fafaaeff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-25T21:40:08.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.964+0000 7fafaaeff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:08.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.964+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7faf84005380 con 0x7fafa406e890 2024-04-25T21:40:08.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.964+0000 7fafa17fa640 1 -- 172.21.15.6:0/3220807399 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 555833778) 0x7faf98015120 con 0x7fafa406e890 2024-04-25T21:40:08.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.964+0000 7fafa17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-25T21:40:08.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.964+0000 7fafa17fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:08.967 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 >> v1:172.21.15.203:6812/3164820986 conn(0x7faf88036b20 legacy=0x7faf88038fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 10 monclient: shutdown 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 >> v1:172.21.15.6:6790/0 conn(0x7fafa406e890 legacy=0x7fafa413beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 shutdown_connections 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 >> 172.21.15.6:0/3220807399 conn(0x7fafa40a5f80 msgr2=0x7fafa409efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 shutdown_connections 2024-04-25T21:40:08.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.968+0000 7fafaaeff640 1 -- 172.21.15.6:0/3220807399 wait complete. 2024-04-25T21:40:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-25T21:40:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-25T21:40:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f15b0136f30 con 0x7f15b006e890 2024-04-25T21:40:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f1597fff640 1 -- 172.21.15.6:0/2667659341 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7f15a8015be0 con 0x7f15b006e890 2024-04-25T21:40:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f1597fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-25T21:40:08.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f1597fff640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:08.975 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-04-25T21:40:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 >> v1:172.21.15.203:6812/3164820986 conn(0x7f159c036bd0 legacy=0x7f159c039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 10 monclient: shutdown 2024-04-25T21:40:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:08.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 >> v1:172.21.15.6:6789/0 conn(0x7f15b006e890 legacy=0x7f15b00a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:08.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 shutdown_connections 2024-04-25T21:40:08.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 >> 172.21.15.6:0/2667659341 conn(0x7f15b00a5f80 msgr2=0x7f15b0136060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:08.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 shutdown_connections 2024-04-25T21:40:08.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:08.972+0000 7f15b61d0640 1 -- 172.21.15.6:0/2667659341 wait complete. 2024-04-25T21:40:08.986 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.3 2024-04-25T21:40:08.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.5 2024-04-25T21:40:09.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T21:40:09.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T21:40:09.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T21:40:09.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T21:40:09.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient: get_monmap_and_config 2024-04-25T21:40:09.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient: build_initial_monmap 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient: monmap: 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:09.973695+0000 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:09.973695+0000 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:09.972 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:09.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 1 Processor -- start 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 1 -- start start 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient: init 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ec06e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ec09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ec13d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec09b0d0 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec0a4ca0 con 0x7fb9ec09bab0 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec06f070 con 0x7fb9ec06e890 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f401a640 10 monclient(hunting): authenticate will time out at 968.883269s 2024-04-25T21:40:09.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f1db6640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fb9ec06e890 0x7fb9ec0a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45734/0 (socket says 172.21.15.6:45734) 2024-04-25T21:40:09.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.968+0000 7fb9f1db6640 1 -- 172.21.15.6:0/926321053 learned_addr learned my addr 172.21.15.6:0/926321053 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f1db6640 1 -- 172.21.15.6:0/926321053 >> v1:172.21.15.6:6790/0 conn(0x7fb9ec06e890 legacy=0x7fb9ec0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f25b7640 1 -- 172.21.15.6:0/926321053 >> v1:172.21.15.6:6789/0 conn(0x7fb9ec13d0a0 legacy=0x7fb9ec13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f15b5640 1 -- 172.21.15.6:0/926321053 >> v1:172.21.15.203:6789/0 conn(0x7fb9ec09bab0 legacy=0x7fb9ec0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157323421 0 0) 0x7fb9e4001830 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): _negotiate 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): my global_id is 4323 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d8003530 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559714881 0 0) 0x7fb9ec06f070 con 0x7fb9ec06e890 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): _negotiate 2024-04-25T21:40:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient(hunting): my global_id is 4304 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9e4001830 con 0x7fb9ec06e890 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 118547034 0 0) 0x7fb9d8003530 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 >> v1:172.21.15.6:6790/0 conn(0x7fb9ec06e890 legacy=0x7fb9ec0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 >> v1:172.21.15.203:6789/0 conn(0x7fb9ec09bab0 legacy=0x7fb9ec0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: _finish_hunting 0 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 monclient: found mon.noname-a 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ec09adf0 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: _finish_auth 0 2024-04-25T21:40:09.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: _check_auth_tickets 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb9e4001e70 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: dump: 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:09.979 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 5 monclient: authenticate success, global_id 4323 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9e40039a0 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 1 -- 172.21.15.6:0/926321053 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb9e4003d30 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 4 set_mon_vals no callback set 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f0db4640 10 monclient: dump: 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.980 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 10 monclient: shutdown 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 1 -- 172.21.15.6:0/926321053 >> v1:172.21.15.6:6789/0 conn(0x7fb9ec13d0a0 legacy=0x7fb9ec13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 1 -- 172.21.15.6:0/926321053 shutdown_connections 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 1 -- 172.21.15.6:0/926321053 >> 172.21.15.6:0/926321053 conn(0x7fb9ec0a5f80 msgr2=0x7fb9ec0a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 1 -- 172.21.15.6:0/926321053 shutdown_connections 2024-04-25T21:40:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.972+0000 7fb9f401a640 1 -- 172.21.15.6:0/926321053 wait complete. 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient: build_initial_monmap 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient: monmap: 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:09.985 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 1 Processor -- start 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 1 -- start start 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient: init 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ec06e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ec09bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ec13d0a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec0a16a0 con 0x7fb9ec09bab0 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec0a35f0 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9ec0a2130 con 0x7fb9ec06e890 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f401a640 10 monclient(hunting): authenticate will time out at 968.888970s 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f15b5640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb9ec09bab0 0x7fb9ec0a0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47710/0 (socket says 172.21.15.6:47710) 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f15b5640 1 -- 172.21.15.6:0/2998851197 learned_addr learned my addr 172.21.15.6:0/2998851197 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f15b5640 1 -- 172.21.15.6:0/2998851197 >> v1:172.21.15.6:6789/0 conn(0x7fb9ec09bab0 legacy=0x7fb9ec0a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f1db6640 1 -- 172.21.15.6:0/2998851197 >> v1:172.21.15.6:6790/0 conn(0x7fb9ec06e890 legacy=0x7fb9ec0a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 784157321 0 0) 0x7fb9ec0a16a0 con 0x7fb9ec09bab0 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): my global_id is 4326 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d0003530 con 0x7fb9ec09bab0 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 222531271 0 0) 0x7fb9ec0a2130 con 0x7fb9ec06e890 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:09.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 10 monclient(hunting): my global_id is 4307 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9ec0a16a0 con 0x7fb9ec06e890 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9f25b7640 1 -- 172.21.15.6:0/2998851197 >> v1:172.21.15.203:6789/0 conn(0x7fb9ec13d0a0 legacy=0x7fb9ec1d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1391044693 0 0) 0x7fb9ec0a16a0 con 0x7fb9ec06e890 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9ec0a2130 con 0x7fb9ec06e890 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.976+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2674800818 0 0) 0x7fb9d0003530 con 0x7fb9ec09bab0 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb9ec0a16a0 con 0x7fb9ec09bab0 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb9e80047f0 con 0x7fb9ec06e890 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb9dc001c10 con 0x7fb9ec09bab0 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1056427876 0 0) 0x7fb9ec0a35f0 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient(hunting): my global_id is 4276 2024-04-25T21:40:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d0003530 con 0x7fb9ec13d0a0 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1444937198 0 0) 0x7fb9ec0a2130 con 0x7fb9ec06e890 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 >> v1:172.21.15.203:6789/0 conn(0x7fb9ec13d0a0 legacy=0x7fb9ec1d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 >> v1:172.21.15.6:6789/0 conn(0x7fb9ec09bab0 legacy=0x7fb9ec0a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 monclient: found mon.c 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ec0a2540 con 0x7fb9ec06e890 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f401a640 5 monclient: authenticate success, global_id 4307 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f401a640 10 monclient: _renew_subs 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f401a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:09.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f401a640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb9ec0a2f70 con 0x7fb9ec06e890 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f401a640 10 monclient: _renew_subs 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f401a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f401a640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb9ec1d55a0 con 0x7fb9ec06e890 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb9e8004c40 con 0x7fb9ec06e890 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fb9e8004f90 con 0x7fb9ec06e890 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 10 monclient: dump: 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e3fff640 4 set_mon_vals no callback set 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9cffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fb9e800fd30 con 0x7fb9ec06e890 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7fb9e80442a0 con 0x7fb9ec06e890 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9cffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c8005090 con 0x7fb9ec06e890 2024-04-25T21:40:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.980+0000 7fb9f15b5640 10 monclient: get_auth_request con 0x7fb9d0036c90 auth_method 0 2024-04-25T21:40:09.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T21:40:09.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T21:40:09.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.988+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fb9e8015620 con 0x7fb9ec06e890 2024-04-25T21:40:09.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.988+0000 7fb9e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:09.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:09.988+0000 7fb9e27fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:10.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient: get_monmap_and_config 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient: monmap: 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:10.024581+0000 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:10.024581+0000 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:10.024 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 1 Processor -- start 2024-04-25T21:40:10.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 1 -- start start 2024-04-25T21:40:10.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient: init 2024-04-25T21:40:10.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient(hunting): picked mon.noname-b con 0x7f30cc06e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient(hunting): picked mon.noname-c con 0x7f30cc09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30c3fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f30cc06e890 0x7f30cc0a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45758/0 (socket says 172.21.15.6:45758) 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d0ea4640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f30cc13d0a0 0x7f30cc13f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47718/0 (socket says 172.21.15.6:47718) 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 10 monclient(hunting): picked mon.noname-a con 0x7f30cc13d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30c3fff640 1 -- 172.21.15.6:0/3194983460 learned_addr learned my addr 172.21.15.6:0/3194983460 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30cbfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f30cc09bab0 0x7f30cc0a4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.6:54500/0 (socket says 172.21.15.6:54500) 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc09b0d0 con 0x7f30cc13d0a0 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc0a4ca0 con 0x7f30cc09bab0 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.020+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc06f070 con 0x7f30cc06e890 2024-04-25T21:40:10.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30d2907640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30d2907640 10 monclient(hunting): authenticate will time out at 968.936158s 2024-04-25T21:40:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30c3fff640 1 -- 172.21.15.6:0/3194983460 >> v1:172.21.15.6:6790/0 conn(0x7f30cc06e890 legacy=0x7f30cc0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cbfff640 1 -- 172.21.15.6:0/3194983460 >> v1:172.21.15.203:6789/0 conn(0x7f30cc09bab0 legacy=0x7f30cc0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30d0ea4640 1 -- 172.21.15.6:0/3194983460 >> v1:172.21.15.6:6789/0 conn(0x7f30cc13d0a0 legacy=0x7f30cc13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176881765 0 0) 0x7f30cc06f070 con 0x7f30cc06e890 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): my global_id is 4310 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30b8003530 con 0x7f30cc06e890 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999973022 0 0) 0x7f30c4001710 con 0x7f30cc13d0a0 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): my global_id is 4329 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30cc06f070 con 0x7f30cc13d0a0 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3114628637 0 0) 0x7f30cc0a4ca0 con 0x7f30cc09bab0 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient(hunting): my global_id is 4279 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30c4001710 con 0x7f30cc09bab0 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1232482219 0 0) 0x7f30b8003530 con 0x7f30cc06e890 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 >> v1:172.21.15.203:6789/0 conn(0x7f30cc09bab0 legacy=0x7f30cc0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 >> v1:172.21.15.6:6789/0 conn(0x7f30cc13d0a0 legacy=0x7f30cc13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 1 monclient: found mon.noname-b 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.024+0000 7f30cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc09adf0 con 0x7f30cc06e890 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f30bc0018d0 con 0x7f30cc06e890 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 5 monclient: authenticate success, global_id 4310 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: dump: 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.034 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30bc001ba0 con 0x7f30cc06e890 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 4 set_mon_vals no callback set 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 1 -- 172.21.15.6:0/3194983460 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f30bc0048b0 con 0x7f30cc06e890 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30cb7fe640 10 monclient: dump: 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 10 monclient: shutdown 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 >> v1:172.21.15.6:6790/0 conn(0x7f30cc06e890 legacy=0x7f30cc0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 shutdown_connections 2024-04-25T21:40:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 >> 172.21.15.6:0/3194983460 conn(0x7f30cc0a5f80 msgr2=0x7f30cc0a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 shutdown_connections 2024-04-25T21:40:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 1 -- 172.21.15.6:0/3194983460 wait complete. 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 10 monclient: monmap: 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:10.042 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 1 Processor -- start 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 1 -- start start 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.028+0000 7f30d2907640 10 monclient: init 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 10 monclient(hunting): picked mon.noname-b con 0x7f30cc06e890 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 10 monclient(hunting): picked mon.noname-a con 0x7f30cc09bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30cbfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f30cc09bab0 0x7f30cc0a1220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47732/0 (socket says 172.21.15.6:47732) 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 10 monclient(hunting): picked mon.noname-c con 0x7f30cc13d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30cbfff640 1 -- 172.21.15.6:0/1313007808 learned_addr learned my addr 172.21.15.6:0/1313007808 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc0a2720 con 0x7f30cc09bab0 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc0a1930 con 0x7f30cc06e890 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30cc09c490 con 0x7f30cc13d0a0 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d2907640 10 monclient(hunting): authenticate will time out at 968.945417s 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30d0ea4640 1 -- 172.21.15.6:0/1313007808 >> v1:172.21.15.6:6790/0 conn(0x7f30cc13d0a0 legacy=0x7f30cc1d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c3fff640 1 -- 172.21.15.6:0/1313007808 >> v1:172.21.15.203:6789/0 conn(0x7f30cc06e890 legacy=0x7f30cc0a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30cbfff640 1 -- 172.21.15.6:0/1313007808 >> v1:172.21.15.6:6789/0 conn(0x7f30cc09bab0 legacy=0x7f30cc0a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3822714840 0 0) 0x7f30cc09c490 con 0x7f30cc13d0a0 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): my global_id is 4313 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30ac003530 con 0x7f30cc13d0a0 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3658030677 0 0) 0x7f30cc0a2720 con 0x7f30cc09bab0 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 10 monclient(hunting): my global_id is 4332 2024-04-25T21:40:10.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.032+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30cc09c490 con 0x7f30cc09bab0 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3004851986 0 0) 0x7f30cc0a1930 con 0x7f30cc06e890 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): my global_id is 4282 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30cc0a2720 con 0x7f30cc06e890 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3265289018 0 0) 0x7f30cc09c490 con 0x7f30cc09bab0 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30cc0a1930 con 0x7f30cc09bab0 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2424212393 0 0) 0x7f30ac003530 con 0x7f30cc13d0a0 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30cc09c490 con 0x7f30cc13d0a0 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f30b4001b80 con 0x7f30cc09bab0 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:10.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): dump: 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f30c40020c0 con 0x7f30cc13d0a0 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient(hunting): dump: 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3813060433 0 0) 0x7f30cc0a1930 con 0x7f30cc09bab0 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 >> v1:172.21.15.6:6790/0 conn(0x7f30cc13d0a0 legacy=0x7f30cc1d3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 >> v1:172.21.15.203:6789/0 conn(0x7f30cc06e890 legacy=0x7f30cc0a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 monclient: found mon.a 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc0a9090 con 0x7f30cc09bab0 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30b4004aa0 con 0x7f30cc09bab0 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30caffd640 4 set_mon_vals no callback set 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f30b4004df0 con 0x7f30cc09bab0 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30d2907640 5 monclient: authenticate success, global_id 4332 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30d2907640 10 monclient: _renew_subs 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30d2907640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30cc138a80 con 0x7f30cc09bab0 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30d2907640 10 monclient: _renew_subs 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30d2907640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30cc13cd70 con 0x7f30cc09bab0 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 10 monclient: dump: 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f30b4005010 con 0x7f30cc09bab0 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.036+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f30b4044130 con 0x7f30cc09bab0 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.040+0000 7f30c3fff640 10 monclient: get_auth_request con 0x7f30ac037190 auth_method 0 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.040+0000 7f30d2907640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.040+0000 7f30d2907640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.040+0000 7f30d2907640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.040+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30cc062e50 con 0x7f30cc09bab0 2024-04-25T21:40:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.048+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f30b4015590 con 0x7f30cc09bab0 2024-04-25T21:40:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.048+0000 7f30c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.048+0000 7f30c97fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient: get_monmap_and_config 2024-04-25T21:40:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient: monmap: 2024-04-25T21:40:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:10.057488+0000 2024-04-25T21:40:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:10.057488+0000 2024-04-25T21:40:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:10.056 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 1 Processor -- start 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 1 -- start start 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient: init 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient(hunting): picked mon.noname-a con 0x7fafa8029640 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient(hunting): picked mon.noname-c con 0x7fafa8023da0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient(hunting): picked mon.noname-b con 0x7fafa80c1c40 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa80276e0 con 0x7fafa8029640 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa80279c0 con 0x7fafa8023da0 2024-04-25T21:40:10.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa8027ca0 con 0x7fafa80c1c40 2024-04-25T21:40:10.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5e33640 10 monclient(hunting): authenticate will time out at 968.967685s 2024-04-25T21:40:10.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5632640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fafa80c1c40 0x7fafa80c4030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45774/0 (socket says 172.21.15.6:45774) 2024-04-25T21:40:10.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb5632640 1 -- 172.21.15.6:0/1322710430 learned_addr learned my addr 172.21.15.6:0/1322710430 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.052+0000 7fafb4e31640 1 --1- 172.21.15.6:0/1322710430 >> v1:172.21.15.6:6789/0 conn(0x7fafa8029640 0x7fafa8025500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47748/0 (socket says 172.21.15.6:47748) 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb4e31640 1 -- 172.21.15.6:0/1322710430 >> v1:172.21.15.6:6789/0 conn(0x7fafa8029640 legacy=0x7fafa8025500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5632640 1 -- 172.21.15.6:0/1322710430 >> v1:172.21.15.6:6790/0 conn(0x7fafa80c1c40 legacy=0x7fafa80c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3339932453 0 0) 0x7fafa80276e0 con 0x7fafa8029640 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): my global_id is 4335 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaffff640 1 -- 172.21.15.6:0/1322710430 >> v1:172.21.15.203:6789/0 conn(0x7fafa8023da0 legacy=0x7fafa80241d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa0003530 con 0x7fafa8029640 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 589154394 0 0) 0x7fafa8027ca0 con 0x7fafa80c1c40 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient(hunting): my global_id is 4316 2024-04-25T21:40:10.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa80276e0 con 0x7fafa80c1c40 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1529157461 0 0) 0x7fafa0003530 con 0x7fafa8029640 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 >> v1:172.21.15.6:6790/0 conn(0x7fafa80c1c40 legacy=0x7fafa80c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 >> v1:172.21.15.203:6789/0 conn(0x7fafa8023da0 legacy=0x7fafa80241d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 monclient: found mon.noname-a 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa8027f80 con 0x7fafa8029640 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafa40018d0 con 0x7fafa8029640 2024-04-25T21:40:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: dump: 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5e33640 5 monclient: authenticate success, global_id 4335 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fafa4001c20 con 0x7fafa8029640 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 1 -- 172.21.15.6:0/1322710430 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafa4004990 con 0x7fafa8029640 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5e33640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5e33640 4 set_mon_vals no callback set 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafaf7fe640 10 monclient: dump: 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.064 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5e33640 10 monclient: shutdown 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.056+0000 7fafb5e33640 1 -- 172.21.15.6:0/1322710430 >> v1:172.21.15.6:6789/0 conn(0x7fafa8029640 legacy=0x7fafa8025500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- 172.21.15.6:0/1322710430 shutdown_connections 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- 172.21.15.6:0/1322710430 >> 172.21.15.6:0/1322710430 conn(0x7fafa8006210 msgr2=0x7fafa8006620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- 172.21.15.6:0/1322710430 shutdown_connections 2024-04-25T21:40:10.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- 172.21.15.6:0/1322710430 wait complete. 2024-04-25T21:40:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient: monmap: 2024-04-25T21:40:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.066 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 Processor -- start 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- start start 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient: init 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient(hunting): picked mon.noname-a con 0x7fafa8023da0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient(hunting): picked mon.noname-b con 0x7fafa8029640 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient(hunting): picked mon.noname-c con 0x7fafa80c1c40 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa8151a00 con 0x7fafa8023da0 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa8020410 con 0x7fafa8029640 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa8020e90 con 0x7fafa80c1c40 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb5e33640 10 monclient(hunting): authenticate will time out at 968.974448s 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb4e31640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fafa8023da0 0x7fafa8150be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47752/0 (socket says 172.21.15.6:47752) 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.060+0000 7fafb4e31640 1 -- 172.21.15.6:0/3930825037 learned_addr learned my addr 172.21.15.6:0/3930825037 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb4e31640 1 -- 172.21.15.6:0/3930825037 >> v1:172.21.15.6:6789/0 conn(0x7fafa8023da0 legacy=0x7fafa8150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5632640 1 -- 172.21.15.6:0/3930825037 >> v1:172.21.15.6:6790/0 conn(0x7fafa80c1c40 legacy=0x7fafa8158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafaffff640 1 -- 172.21.15.6:0/3930825037 >> v1:172.21.15.203:6789/0 conn(0x7fafa8029640 legacy=0x7fafa81512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818669119 0 0) 0x7fafa8151a00 con 0x7fafa8023da0 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): my global_id is 4338 2024-04-25T21:40:10.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf94003530 con 0x7fafa8023da0 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047359886 0 0) 0x7fafa8020410 con 0x7fafa8029640 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): my global_id is 4288 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa8151a00 con 0x7fafa8029640 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940898631 0 0) 0x7fafa8020e90 con 0x7fafa80c1c40 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): my global_id is 4319 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa8020410 con 0x7fafa80c1c40 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 308285994 0 0) 0x7faf94003530 con 0x7fafa8023da0 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafa8020e90 con 0x7fafa8023da0 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafa4004990 con 0x7fafa8023da0 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.071 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4118201031 0 0) 0x7fafa8020410 con 0x7fafa80c1c40 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faf94003530 con 0x7fafa80c1c40 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafb004fa80 con 0x7fafa80c1c40 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.072 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1141894896 0 0) 0x7fafa8151a00 con 0x7fafa8029640 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafa8020410 con 0x7fafa8029640 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7faf9c001b40 con 0x7fafa8029640 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.073 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1837025958 0 0) 0x7fafa8020e90 con 0x7fafa8023da0 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 >> v1:172.21.15.6:6790/0 conn(0x7fafa80c1c40 legacy=0x7fafa8158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 >> v1:172.21.15.203:6789/0 conn(0x7fafa8029640 legacy=0x7fafa81512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 monclient: found mon.a 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa80b4260 con 0x7fafa8023da0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 5 monclient: authenticate success, global_id 4338 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 10 monclient: _renew_subs 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fafa80b4540 con 0x7fafa8023da0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 10 monclient: _renew_subs 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fafa80b3c40 con 0x7fafa8023da0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fafa4001c20 con 0x7fafa8023da0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafa4005120 con 0x7fafa8023da0 2024-04-25T21:40:10.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 10 monclient: dump: 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fafa400fec0 con 0x7fafa8023da0 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafaeffd640 4 set_mon_vals no callback set 2024-04-25T21:40:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafb5e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.064+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7fafa40443f0 con 0x7fafa8023da0 2024-04-25T21:40:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.068+0000 7fafb5e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.068+0000 7fafb5e33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.068+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa8005130 con 0x7fafa8023da0 2024-04-25T21:40:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.068+0000 7fafaffff640 10 monclient: get_auth_request con 0x7faf94036db0 auth_method 0 2024-04-25T21:40:10.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fafa40193c0 con 0x7fafa8023da0 2024-04-25T21:40:10.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7fafad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7fafad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient: get_monmap_and_config 2024-04-25T21:40:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient: monmap: 2024-04-25T21:40:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:10.081666+0000 2024-04-25T21:40:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:10.081666+0000 2024-04-25T21:40:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:10.080 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 1 Processor -- start 2024-04-25T21:40:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 1 -- start start 2024-04-25T21:40:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient: init 2024-04-25T21:40:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient(hunting): picked mon.noname-b con 0x7f688c06e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient(hunting): picked mon.noname-c con 0x7f688c09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient(hunting): picked mon.noname-a con 0x7f688c13d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688c09b0d0 con 0x7f688c13d0a0 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f6891eca640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f688c13d0a0 0x7f688c13f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47756/0 (socket says 172.21.15.6:47756) 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f6891eca640 1 -- 172.21.15.6:0/5242619 learned_addr learned my addr 172.21.15.6:0/5242619 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 1 -- 172.21.15.6:0/5242619 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688c0a4ca0 con 0x7f688c09bab0 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 1 -- 172.21.15.6:0/5242619 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688c06f070 con 0x7f688c06e890 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.076+0000 7f689392d640 10 monclient(hunting): authenticate will time out at 968.991985s 2024-04-25T21:40:10.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6891eca640 1 -- 172.21.15.6:0/5242619 >> v1:172.21.15.6:6789/0 conn(0x7f688c13d0a0 legacy=0x7f688c13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f68916c9640 1 -- 172.21.15.6:0/5242619 >> v1:172.21.15.6:6790/0 conn(0x7f688c06e890 legacy=0x7f688c0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6890ec8640 1 -- 172.21.15.6:0/5242619 >> v1:172.21.15.203:6789/0 conn(0x7f688c09bab0 legacy=0x7f688c0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971768894 0 0) 0x7f68840017e0 con 0x7f688c13d0a0 2024-04-25T21:40:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): my global_id is 4341 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6878003530 con 0x7f688c13d0a0 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1027435318 0 0) 0x7f688c06f070 con 0x7f688c06e890 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): my global_id is 4322 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68840017e0 con 0x7f688c06e890 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700402403 0 0) 0x7f688c0a4ca0 con 0x7f688c09bab0 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient(hunting): my global_id is 4291 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f688c06f070 con 0x7f688c09bab0 2024-04-25T21:40:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1539828063 0 0) 0x7f68840017e0 con 0x7f688c06e890 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 >> v1:172.21.15.203:6789/0 conn(0x7f688c09bab0 legacy=0x7f688c0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 >> v1:172.21.15.6:6789/0 conn(0x7f688c13d0a0 legacy=0x7f688c13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 monclient: found mon.noname-b 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688c09adf0 con 0x7f688c06e890 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6888001920 con 0x7f688c06e890 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: dump: 2024-04-25T21:40:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f689392d640 5 monclient: authenticate success, global_id 4322 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f689392d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6888001c70 con 0x7f688c06e890 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 1 -- 172.21.15.6:0/5242619 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f68880037c0 con 0x7f688c06e890 2024-04-25T21:40:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f6883fff640 10 monclient: dump: 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f689392d640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.080+0000 7f689392d640 4 set_mon_vals no callback set 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient: shutdown 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 1 -- 172.21.15.6:0/5242619 >> v1:172.21.15.6:6790/0 conn(0x7f688c06e890 legacy=0x7f688c0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 1 -- 172.21.15.6:0/5242619 shutdown_connections 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 1 -- 172.21.15.6:0/5242619 >> 172.21.15.6:0/5242619 conn(0x7f688c0a5f80 msgr2=0x7f688c0a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 1 -- 172.21.15.6:0/5242619 shutdown_connections 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 1 -- 172.21.15.6:0/5242619 wait complete. 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient: monmap: 2024-04-25T21:40:10.092 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 1 Processor -- start 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 1 -- start start 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient: init 2024-04-25T21:40:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient(hunting): picked mon.noname-b con 0x7f688c06e890 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.084+0000 7f689392d640 10 monclient(hunting): picked mon.noname-c con 0x7f688c09bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6890ec8640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f688c09bab0 0x7f688c1304b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45798/0 (socket says 172.21.15.6:45798) 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6891eca640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f688c13d0a0 0x7f688c1d4090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47758/0 (socket says 172.21.15.6:47758) 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 10 monclient(hunting): picked mon.noname-a con 0x7f688c13d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6890ec8640 1 -- 172.21.15.6:0/2711386198 learned_addr learned my addr 172.21.15.6:0/2711386198 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688c0a1760 con 0x7f688c13d0a0 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688c0a1a40 con 0x7f688c06e890 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688c13ff70 con 0x7f688c09bab0 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 10 monclient(hunting): authenticate will time out at 969.000580s 2024-04-25T21:40:10.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6890ec8640 1 -- 172.21.15.6:0/2711386198 >> v1:172.21.15.6:6790/0 conn(0x7f688c09bab0 legacy=0x7f688c1304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6891eca640 1 -- 172.21.15.6:0/2711386198 >> v1:172.21.15.6:6789/0 conn(0x7f688c13d0a0 legacy=0x7f688c1d4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f68916c9640 1 -- 172.21.15.6:0/2711386198 >> v1:172.21.15.203:6789/0 conn(0x7f688c06e890 legacy=0x7f688c12fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1636447664 0 0) 0x7f688c13ff70 con 0x7f688c09bab0 2024-04-25T21:40:10.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): my global_id is 4325 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6870003530 con 0x7f688c09bab0 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146716369 0 0) 0x7f688c0a1a40 con 0x7f688c06e890 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): my global_id is 4294 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f688c13ff70 con 0x7f688c06e890 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341894997 0 0) 0x7f688c0a1760 con 0x7f688c13d0a0 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): my global_id is 4344 2024-04-25T21:40:10.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f688c0a1a40 con 0x7f688c13d0a0 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2918632923 0 0) 0x7f6870003530 con 0x7f688c09bab0 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f688c0a1760 con 0x7f688c09bab0 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f687c001ba0 con 0x7f688c09bab0 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): dump: 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.098 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3547079554 0 0) 0x7f688c0a1760 con 0x7f688c09bab0 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 >> v1:172.21.15.203:6789/0 conn(0x7f688c06e890 legacy=0x7f688c12fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 >> v1:172.21.15.6:6789/0 conn(0x7f688c13d0a0 legacy=0x7f688c1d4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 monclient: found mon.c 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688c09c490 con 0x7f688c09bab0 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f687c0038d0 con 0x7f688c09bab0 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f687c003c20 con 0x7f688c09bab0 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f6881ffb640 10 monclient: dump: 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.100 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f68837fe640 4 set_mon_vals no callback set 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 5 monclient: authenticate success, global_id 4325 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 10 monclient: _renew_subs 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f688c140190 con 0x7f688c09bab0 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 10 monclient: _renew_subs 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.088+0000 7f689392d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f688c13fcf0 con 0x7f688c09bab0 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f687c003e80 con 0x7f688c09bab0 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f689392d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f689392d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f689392d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f688c062e50 con 0x7f688c09bab0 2024-04-25T21:40:10.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f687c044360 con 0x7f688c09bab0 2024-04-25T21:40:10.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.092+0000 7f68916c9640 10 monclient: get_auth_request con 0x7f6870036da0 auth_method 0 2024-04-25T21:40:10.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.100+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f687c015740 con 0x7f688c09bab0 2024-04-25T21:40:10.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.100+0000 7f6881ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.100+0000 7f6881ffb640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.120+0000 7f58c3101640 10 monclient: get_monmap_and_config 2024-04-25T21:40:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.120+0000 7f58c3101640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.120+0000 7f58c3101640 10 monclient: monmap: 2024-04-25T21:40:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:10.126612+0000 2024-04-25T21:40:10.124 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:10.126612+0000 2024-04-25T21:40:10.125 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.125 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.125 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.125 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:10.125 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:10.125 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.120+0000 7f58c3101640 1 Processor -- start 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.120+0000 7f58c3101640 1 -- start start 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.120+0000 7f58c3101640 10 monclient: init 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient(hunting): picked mon.noname-c con 0x7f58bc06e890 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc09bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc13d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c169e640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f58bc13d0a0 0x7f58bc13f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45800/0 (socket says 172.21.15.6:45800) 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c169e640 1 -- 172.21.15.6:0/608781489 learned_addr learned my addr 172.21.15.6:0/608781489 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc09b0d0 con 0x7f58bc09bab0 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 1 -- 172.21.15.6:0/608781489 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc0a4ca0 con 0x7f58bc06e890 2024-04-25T21:40:10.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 1 -- 172.21.15.6:0/608781489 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc06f070 con 0x7f58bc13d0a0 2024-04-25T21:40:10.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient(hunting): authenticate will time out at 969.036698s 2024-04-25T21:40:10.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c169e640 1 -- 172.21.15.6:0/608781489 >> v1:172.21.15.6:6790/0 conn(0x7f58bc13d0a0 legacy=0x7f58bc13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bbfff640 1 -- 172.21.15.6:0/608781489 >> v1:172.21.15.6:6789/0 conn(0x7f58bc09bab0 legacy=0x7f58bc0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3255634734 0 0) 0x7f58a80014b0 con 0x7f58bc09bab0 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): my global_id is 4347 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58ac003530 con 0x7f58bc09bab0 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048687378 0 0) 0x7f58bc06f070 con 0x7f58bc13d0a0 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient(hunting): my global_id is 4328 2024-04-25T21:40:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58a80014b0 con 0x7f58bc13d0a0 2024-04-25T21:40:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4130757000 0 0) 0x7f58ac003530 con 0x7f58bc09bab0 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 >> v1:172.21.15.6:6790/0 conn(0x7f58bc13d0a0 legacy=0x7f58bc13f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 >> v1:172.21.15.203:6789/0 conn(0x7f58bc06e890 legacy=0x7f58bc0a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 monclient: found mon.noname-a 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc09adf0 con 0x7f58bc09bab0 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f58a8001af0 con 0x7f58bc09bab0 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: dump: 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 5 monclient: authenticate success, global_id 4347 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58a8003620 con 0x7f58bc09bab0 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 1 -- 172.21.15.6:0/608781489 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f58a80039b0 con 0x7f58bc09bab0 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58bb7fe640 10 monclient: dump: 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.124+0000 7f58c3101640 4 set_mon_vals no callback set 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient: shutdown 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/608781489 >> v1:172.21.15.6:6789/0 conn(0x7f58bc09bab0 legacy=0x7f58bc0a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/608781489 shutdown_connections 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/608781489 >> 172.21.15.6:0/608781489 conn(0x7f58bc0a5f80 msgr2=0x7f58bc0a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/608781489 shutdown_connections 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/608781489 wait complete. 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient: monmap: 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.134 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:10.135 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.135 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:10.135 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:10.135 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 Processor -- start 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- start start 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient: init 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc06e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc13d0a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c0e9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f58bc06e890 0x7f58bc12fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47776/0 (socket says 172.21.15.6:47776) 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c0e9d640 1 -- 172.21.15.6:0/3358111600 learned_addr learned my addr 172.21.15.6:0/3358111600 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient(hunting): picked mon.noname-c con 0x7f58bc13ea90 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc0a1960 con 0x7f58bc06e890 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc13fb00 con 0x7f58bc13d0a0 2024-04-25T21:40:10.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc0a1c40 con 0x7f58bc13ea90 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.128+0000 7f58c3101640 10 monclient(hunting): authenticate will time out at 969.043350s 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c169e640 1 -- 172.21.15.6:0/3358111600 >> v1:172.21.15.6:6790/0 conn(0x7f58bc13ea90 legacy=0x7f58bc1d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c0e9d640 1 -- 172.21.15.6:0/3358111600 >> v1:172.21.15.6:6789/0 conn(0x7f58bc06e890 legacy=0x7f58bc12fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395823463 0 0) 0x7f58bc0a1c40 con 0x7f58bc13ea90 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): my global_id is 4331 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58a0003530 con 0x7f58bc13ea90 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1271122595 0 0) 0x7f58bc0a1960 con 0x7f58bc06e890 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): my global_id is 4350 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58bc0a1c40 con 0x7f58bc06e890 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58bbfff640 1 -- 172.21.15.6:0/3358111600 >> v1:172.21.15.203:6789/0 conn(0x7f58bc13d0a0 legacy=0x7f58bc1d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3675309795 0 0) 0x7f58a0003530 con 0x7f58bc13ea90 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58bc0a1960 con 0x7f58bc13ea90 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f58b4001fc0 con 0x7f58bc13ea90 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): dump: 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.140 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839449657 0 0) 0x7f58bc13fb00 con 0x7f58bc13d0a0 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): my global_id is 4297 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58a0003530 con 0x7f58bc13d0a0 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1337934422 0 0) 0x7f58bc0a1c40 con 0x7f58bc06e890 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58bc13fb00 con 0x7f58bc06e890 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f58b00042c0 con 0x7f58bc06e890 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient(hunting): dump: 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1656053338 0 0) 0x7f58bc0a1960 con 0x7f58bc13ea90 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 >> v1:172.21.15.203:6789/0 conn(0x7f58bc13d0a0 legacy=0x7f58bc1d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 >> v1:172.21.15.6:6789/0 conn(0x7f58bc06e890 legacy=0x7f58bc12fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 monclient: found mon.c 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc0a25b0 con 0x7f58bc13ea90 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c3101640 5 monclient: authenticate success, global_id 4331 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c3101640 10 monclient: _renew_subs 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58b4004e20 con 0x7f58bc13ea90 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c3101640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f58bc0a1680 con 0x7f58bc13ea90 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c3101640 10 monclient: _renew_subs 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c3101640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f58bc130540 con 0x7f58bc13ea90 2024-04-25T21:40:10.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f58b4005170 con 0x7f58bc13ea90 2024-04-25T21:40:10.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58b97fa640 10 monclient: dump: 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.132+0000 7f58baffd640 4 set_mon_vals no callback set 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.136+0000 7f58c3101640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.136+0000 7f58c3101640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.136+0000 7f58c3101640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.136+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f589c005090 con 0x7f58bc13ea90 2024-04-25T21:40:10.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.136+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f58b400ff10 con 0x7f58bc13ea90 2024-04-25T21:40:10.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.136+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f58b40445b0 con 0x7f58bc13ea90 2024-04-25T21:40:10.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.136+0000 7f58c0e9d640 10 monclient: get_auth_request con 0x7f58a0036c20 auth_method 0 2024-04-25T21:40:10.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.140+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f58b40159b0 con 0x7f58bc13ea90 2024-04-25T21:40:10.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.140+0000 7f58b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.140+0000 7f58b97fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.140+0000 7f40d5aff640 10 monclient: get_monmap_and_config 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.140+0000 7f40d5aff640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.140+0000 7f40d5aff640 10 monclient: monmap: 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:10.147273+0000 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:10.147273+0000 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 1 Processor -- start 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 1 -- start start 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient: init 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient(hunting): picked mon.noname-b con 0x7f40d006e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient(hunting): picked mon.noname-c con 0x7f40d009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40cf7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f40d006e890 0x7f40d00a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45808/0 (socket says 172.21.15.6:45808) 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40cf7fe640 1 -- 172.21.15.6:0/4113374742 learned_addr learned my addr 172.21.15.6:0/4113374742 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient(hunting): picked mon.noname-a con 0x7f40d013d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d009b0d0 con 0x7f40d013d0a0 2024-04-25T21:40:10.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d00a4ca0 con 0x7f40d009bab0 2024-04-25T21:40:10.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d006f070 con 0x7f40d006e890 2024-04-25T21:40:10.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.144+0000 7f40d5aff640 10 monclient(hunting): authenticate will time out at 969.058235s 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40cf7fe640 1 -- 172.21.15.6:0/4113374742 >> v1:172.21.15.6:6790/0 conn(0x7f40d006e890 legacy=0x7f40d00a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40cffff640 1 -- 172.21.15.6:0/4113374742 >> v1:172.21.15.6:6789/0 conn(0x7f40d013d0a0 legacy=0x7f40d013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ceffd640 1 -- 172.21.15.6:0/4113374742 >> v1:172.21.15.203:6789/0 conn(0x7f40d009bab0 legacy=0x7f40d00a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456326078 0 0) 0x7f40d006f070 con 0x7f40d006e890 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): my global_id is 4334 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40b8003530 con 0x7f40d006e890 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3633624883 0 0) 0x7f40c8001830 con 0x7f40d013d0a0 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): my global_id is 4353 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40d006f070 con 0x7f40d013d0a0 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28240860 0 0) 0x7f40d00a4ca0 con 0x7f40d009bab0 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient(hunting): my global_id is 4300 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40c8001830 con 0x7f40d009bab0 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3368533055 0 0) 0x7f40d006f070 con 0x7f40d013d0a0 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 >> v1:172.21.15.6:6790/0 conn(0x7f40d006e890 legacy=0x7f40d00a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 >> v1:172.21.15.203:6789/0 conn(0x7f40d009bab0 legacy=0x7f40d00a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 monclient: found mon.noname-a 2024-04-25T21:40:10.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d009adf0 con 0x7f40d013d0a0 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f40c8001df0 con 0x7f40d013d0a0 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: dump: 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.154 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 5 monclient: authenticate success, global_id 4353 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40c80020c0 con 0x7f40d013d0a0 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 1 -- 172.21.15.6:0/4113374742 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f40c8003b40 con 0x7f40d013d0a0 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40ce7fc640 10 monclient: dump: 2024-04-25T21:40:10.155 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 4 set_mon_vals no callback set 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 10 monclient: shutdown 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 >> v1:172.21.15.6:6789/0 conn(0x7f40d013d0a0 legacy=0x7f40d013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 shutdown_connections 2024-04-25T21:40:10.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 >> 172.21.15.6:0/4113374742 conn(0x7f40d00a5f80 msgr2=0x7f40d00a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 shutdown_connections 2024-04-25T21:40:10.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 1 -- 172.21.15.6:0/4113374742 wait complete. 2024-04-25T21:40:10.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.148+0000 7f40d5aff640 10 monclient: monmap: 2024-04-25T21:40:10.163 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.163 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 1 Processor -- start 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 1 -- start start 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient: init 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient(hunting): picked mon.noname-c con 0x7f40d006e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient(hunting): picked mon.noname-b con 0x7f40d009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40cf7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f40d006e890 0x7f40d01cbc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45820/0 (socket says 172.21.15.6:45820) 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40cffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f40d013d0a0 0x7f40d01d3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47790/0 (socket says 172.21.15.6:47790) 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient(hunting): picked mon.noname-a con 0x7f40d013d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40cf7fe640 1 -- 172.21.15.6:0/2465604407 learned_addr learned my addr 172.21.15.6:0/2465604407 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d01ccab0 con 0x7f40d013d0a0 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d00a90d0 con 0x7f40d009bab0 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d00a1890 con 0x7f40d006e890 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40d5aff640 10 monclient(hunting): authenticate will time out at 969.065995s 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40cf7fe640 1 -- 172.21.15.6:0/2465604407 >> v1:172.21.15.6:6790/0 conn(0x7f40d006e890 legacy=0x7f40d01cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.152+0000 7f40cffff640 1 -- 172.21.15.6:0/2465604407 >> v1:172.21.15.6:6789/0 conn(0x7f40d013d0a0 legacy=0x7f40d01d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40ceffd640 1 -- 172.21.15.6:0/2465604407 >> v1:172.21.15.203:6789/0 conn(0x7f40d009bab0 legacy=0x7f40d01cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538098103 0 0) 0x7f40d00a1890 con 0x7f40d006e890 2024-04-25T21:40:10.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): my global_id is 4337 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40bc003530 con 0x7f40d006e890 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861174448 0 0) 0x7f40d01ccab0 con 0x7f40d013d0a0 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): my global_id is 4356 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40d00a1890 con 0x7f40d013d0a0 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 949407222 0 0) 0x7f40d00a90d0 con 0x7f40d009bab0 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): my global_id is 4303 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40d01ccab0 con 0x7f40d009bab0 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1385065418 0 0) 0x7f40bc003530 con 0x7f40d006e890 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40d00a90d0 con 0x7f40d006e890 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f40c0001c30 con 0x7f40d006e890 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): dump: 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.167 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3432836054 0 0) 0x7f40d00a1890 con 0x7f40d013d0a0 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40bc003530 con 0x7f40d013d0a0 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f40c80039a0 con 0x7f40d013d0a0 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): dump: 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.168 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1503720892 0 0) 0x7f40d01ccab0 con 0x7f40d009bab0 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f40d00a1890 con 0x7f40d009bab0 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f40c4001af0 con 0x7f40d009bab0 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient(hunting): dump: 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.169 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3240360718 0 0) 0x7f40d00a90d0 con 0x7f40d006e890 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 >> v1:172.21.15.203:6789/0 conn(0x7f40d009bab0 legacy=0x7f40d01cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 >> v1:172.21.15.6:6789/0 conn(0x7f40d013d0a0 legacy=0x7f40d01d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 monclient: found mon.c 2024-04-25T21:40:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d00a21d0 con 0x7f40d006e890 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 5 monclient: authenticate success, global_id 4337 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 10 monclient: _renew_subs 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f40c0004b60 con 0x7f40d006e890 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f40d013fb60 con 0x7f40d006e890 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f40c0004eb0 con 0x7f40d006e890 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 10 monclient: dump: 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40cdffb640 4 set_mon_vals no callback set 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 10 monclient: _renew_subs 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f40d00a3360 con 0x7f40d006e890 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d013fd90 con 0x7f40d006e890 2024-04-25T21:40:10.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.156+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f40c000fcd0 con 0x7f40d006e890 2024-04-25T21:40:10.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.160+0000 7f40ceffd640 10 monclient: get_auth_request con 0x7f40bc036d30 auth_method 0 2024-04-25T21:40:10.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.168+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f40c0043070 con 0x7f40d006e890 2024-04-25T21:40:10.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.168+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f40c0015520 con 0x7f40d006e890 2024-04-25T21:40:10.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.168+0000 7f40b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.168+0000 7f40b7fff640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.196+0000 7fb9cffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-25T21:40:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.196+0000 7fb9cffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-25T21:40:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.196+0000 7fb9cffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.196+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb9c8005380 con 0x7fb9ec06e890 2024-04-25T21:40:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.196+0000 7fb9e27fc640 1 -- 172.21.15.6:0/2998851197 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 203001719) 0x7fb9e8014f20 con 0x7fb9ec06e890 2024-04-25T21:40:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.196+0000 7fb9e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-25T21:40:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.196+0000 7fb9e27fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:10.201 INFO:teuthology.orchestra.run.smithi006.stdout:51539607556 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 >> v1:172.21.15.203:6812/3164820986 conn(0x7fb9d0036c90 legacy=0x7fb9d0039150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 10 monclient: shutdown 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 >> v1:172.21.15.6:6790/0 conn(0x7fb9ec06e890 legacy=0x7fb9ec0a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 shutdown_connections 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 >> 172.21.15.6:0/2998851197 conn(0x7fb9ec0a5f80 msgr2=0x7fb9ec1361f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 shutdown_connections 2024-04-25T21:40:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.200+0000 7fb9cffff640 1 -- 172.21.15.6:0/2998851197 wait complete. 2024-04-25T21:40:10.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607556 for osd.0 2024-04-25T21:40:10.218 DEBUG:teuthology.parallel:result is None 2024-04-25T21:40:10.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.284+0000 7f689392d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-25T21:40:10.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.284+0000 7f689392d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-25T21:40:10.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.284+0000 7f689392d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.284+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f688c137250 con 0x7f688c09bab0 2024-04-25T21:40:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f6881ffb640 1 -- 172.21.15.6:0/2711386198 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 956180462) 0x7f687c015040 con 0x7f688c09bab0 2024-04-25T21:40:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f6881ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-25T21:40:10.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f6881ffb640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:10.290 INFO:teuthology.orchestra.run.smithi006.stdout:51539607555 2024-04-25T21:40:10.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 >> v1:172.21.15.203:6812/3164820986 conn(0x7f6870036da0 legacy=0x7f6870039260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 10 monclient: shutdown 2024-04-25T21:40:10.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 >> v1:172.21.15.6:6790/0 conn(0x7f688c09bab0 legacy=0x7f688c1304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 shutdown_connections 2024-04-25T21:40:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 >> 172.21.15.6:0/2711386198 conn(0x7f688c0a5f80 msgr2=0x7f688c136380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 shutdown_connections 2024-04-25T21:40:10.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.288+0000 7f689392d640 1 -- 172.21.15.6:0/2711386198 wait complete. 2024-04-25T21:40:10.306 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607555 for osd.2 2024-04-25T21:40:10.306 DEBUG:teuthology.parallel:result is None 2024-04-25T21:40:10.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-25T21:40:10.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-25T21:40:10.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f30cc137050 con 0x7f30cc09bab0 2024-04-25T21:40:10.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30c97fa640 1 -- 172.21.15.6:0/1313007808 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 203001719) 0x7f30b4014e90 con 0x7f30cc09bab0 2024-04-25T21:40:10.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-25T21:40:10.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30c97fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:10.367 INFO:teuthology.orchestra.run.smithi006.stdout:51539607556 2024-04-25T21:40:10.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 >> v1:172.21.15.203:6812/3164820986 conn(0x7f30ac037190 legacy=0x7f30ac039650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 10 monclient: shutdown 2024-04-25T21:40:10.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.364+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 >> v1:172.21.15.6:6789/0 conn(0x7f30cc09bab0 legacy=0x7f30cc0a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.368+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 shutdown_connections 2024-04-25T21:40:10.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.368+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 >> 172.21.15.6:0/1313007808 conn(0x7f30cc0a5f80 msgr2=0x7f30cc136180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.368+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 shutdown_connections 2024-04-25T21:40:10.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.368+0000 7f30d2907640 1 -- 172.21.15.6:0/1313007808 wait complete. 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f589c005380 con 0x7f58bc13ea90 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58b97fa640 1 -- 172.21.15.6:0/3358111600 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 532354816) 0x7f58b4019660 con 0x7f58bc13ea90 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58b97fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:10.383 INFO:teuthology.orchestra.run.smithi006.stdout:51539607557 2024-04-25T21:40:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 >> v1:172.21.15.203:6812/3164820986 conn(0x7f58a0036c20 legacy=0x7f58a00390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 10 monclient: shutdown 2024-04-25T21:40:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 >> v1:172.21.15.6:6790/0 conn(0x7f58bc13ea90 legacy=0x7f58bc1d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 shutdown_connections 2024-04-25T21:40:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 >> 172.21.15.6:0/3358111600 conn(0x7f58bc0a5f80 msgr2=0x7f58bc136380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.380+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 shutdown_connections 2024-04-25T21:40:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.384+0000 7f58c3101640 1 -- 172.21.15.6:0/3358111600 wait complete. 2024-04-25T21:40:10.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607556 for osd.1 2024-04-25T21:40:10.389 DEBUG:teuthology.parallel:result is None 2024-04-25T21:40:10.400 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607557 for osd.5 2024-04-25T21:40:10.400 DEBUG:teuthology.parallel:result is None 2024-04-25T21:40:10.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-25T21:40:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-25T21:40:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fafa80c1a20 con 0x7fafa8023da0 2024-04-25T21:40:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafad7fa640 1 -- 172.21.15.6:0/3930825037 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 203001719) 0x7fafa401dc80 con 0x7fafa8023da0 2024-04-25T21:40:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-25T21:40:10.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafad7fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:10.402 INFO:teuthology.orchestra.run.smithi006.stdout:51539607556 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 >> v1:172.21.15.203:6812/3164820986 conn(0x7faf94036db0 legacy=0x7faf94039270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 10 monclient: shutdown 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 >> v1:172.21.15.6:6789/0 conn(0x7fafa8023da0 legacy=0x7fafa8150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 shutdown_connections 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 >> 172.21.15.6:0/3930825037 conn(0x7fafa8006210 msgr2=0x7fafa80065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 shutdown_connections 2024-04-25T21:40:10.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7fafb5e33640 1 -- 172.21.15.6:0/3930825037 wait complete. 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7f40d5aff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7f40d5aff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7f40d5aff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f40d0062e50 con 0x7f40d006e890 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7f40b7fff640 1 -- 172.21.15.6:0/2465604407 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 1444633209) 0x7f40c0014e20 con 0x7f40d006e890 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7f40b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.400+0000 7f40b7fff640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:10.405 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2024-04-25T21:40:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 >> v1:172.21.15.203:6812/3164820986 conn(0x7f40bc036d30 legacy=0x7f40bc0391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 10 monclient: shutdown 2024-04-25T21:40:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 >> v1:172.21.15.6:6790/0 conn(0x7f40d006e890 legacy=0x7f40d01cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 shutdown_connections 2024-04-25T21:40:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 >> 172.21.15.6:0/2465604407 conn(0x7f40d00a5f80 msgr2=0x7f40d0136220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 shutdown_connections 2024-04-25T21:40:10.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.404+0000 7f40d5aff640 1 -- 172.21.15.6:0/2465604407 wait complete. 2024-04-25T21:40:10.420 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607555 got 51539607556 for osd.4 2024-04-25T21:40:10.420 DEBUG:teuthology.parallel:result is None 2024-04-25T21:40:10.422 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-04-25T21:40:10.422 DEBUG:teuthology.parallel:result is None 2024-04-25T21:40:10.422 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T21:40:10.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.568+0000 7f94741e9640 10 monclient: get_monmap_and_config 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.568+0000 7f94741e9640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.568+0000 7f94741e9640 10 monclient: monmap: 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:10.573966+0000 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:10.573966+0000 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:10.572 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.568+0000 7f94741e9640 1 Processor -- start 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.568+0000 7f94741e9640 1 -- start start 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.568+0000 7f94741e9640 10 monclient: init 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient(hunting): picked mon.noname-b con 0x7f946c0a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient(hunting): picked mon.noname-a con 0x7f946c0a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient(hunting): picked mon.noname-c con 0x7f946c09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c09c590 con 0x7f946c0a4970 2024-04-25T21:40:10.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c09adf0 con 0x7f946c09bab0 2024-04-25T21:40:10.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c09bf20 con 0x7f946c0a3e80 2024-04-25T21:40:10.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient(hunting): authenticate will time out at 969.484309s 2024-04-25T21:40:10.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9471f85640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f946c0a3e80 0x7f946c0a4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45836/0 (socket says 172.21.15.6:45836) 2024-04-25T21:40:10.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9471f85640 1 -- 172.21.15.6:0/2648267823 learned_addr learned my addr 172.21.15.6:0/2648267823 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9471784640 1 -- 172.21.15.6:0/2648267823 >> v1:172.21.15.6:6789/0 conn(0x7f946c0a4970 legacy=0x7f946c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9471f85640 1 -- 172.21.15.6:0/2648267823 >> v1:172.21.15.6:6790/0 conn(0x7f946c0a3e80 legacy=0x7f946c0a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1022284074 0 0) 0x7f946c09bf20 con 0x7f946c0a3e80 2024-04-25T21:40:10.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): my global_id is 4340 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9458003530 con 0x7f946c0a3e80 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302412865 0 0) 0x7f946c09c590 con 0x7f946c0a4970 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient(hunting): my global_id is 4359 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f946c09bf20 con 0x7f946c0a4970 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9472786640 1 -- 172.21.15.6:0/2648267823 >> v1:172.21.15.203:6789/0 conn(0x7f946c09bab0 legacy=0x7f946c13f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1284457100 0 0) 0x7f9458003530 con 0x7f946c0a3e80 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 >> v1:172.21.15.203:6789/0 conn(0x7f946c09bab0 legacy=0x7f946c13f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 >> v1:172.21.15.6:6789/0 conn(0x7f946c0a4970 legacy=0x7f946c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 monclient: found mon.noname-b 2024-04-25T21:40:10.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c06eaf0 con 0x7f946c0a3e80 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 5 monclient: authenticate success, global_id 4340 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9468001c50 con 0x7f946c0a3e80 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: dump: 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.580 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9468001fa0 con 0x7f946c0a3e80 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 1 -- 172.21.15.6:0/2648267823 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9468004dd0 con 0x7f946c0a3e80 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 4 set_mon_vals no callback set 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f9470f83640 10 monclient: dump: 2024-04-25T21:40:10.581 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 10 monclient: shutdown 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 1 -- 172.21.15.6:0/2648267823 >> v1:172.21.15.6:6790/0 conn(0x7f946c0a3e80 legacy=0x7f946c0a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 1 -- 172.21.15.6:0/2648267823 shutdown_connections 2024-04-25T21:40:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.572+0000 7f94741e9640 1 -- 172.21.15.6:0/2648267823 >> 172.21.15.6:0/2648267823 conn(0x7f946c0a5f80 msgr2=0x7f946c0a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 1 -- 172.21.15.6:0/2648267823 shutdown_connections 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 1 -- 172.21.15.6:0/2648267823 wait complete. 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient: monmap: 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 1 Processor -- start 2024-04-25T21:40:10.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 1 -- start start 2024-04-25T21:40:10.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient: init 2024-04-25T21:40:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient(hunting): picked mon.noname-b con 0x7f946c09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient(hunting): picked mon.noname-a con 0x7f946c0a3e80 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient(hunting): picked mon.noname-c con 0x7f946c0a4970 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c0a5c80 con 0x7f946c0a3e80 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c139c70 con 0x7f946c09bab0 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c13ee10 con 0x7f946c0a4970 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94741e9640 10 monclient(hunting): authenticate will time out at 969.489616s 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f9472786640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f946c0a4970 0x7f946c1d4000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45844/0 (socket says 172.21.15.6:45844) 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f9472786640 1 -- 172.21.15.6:0/740872600 learned_addr learned my addr 172.21.15.6:0/740872600 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f9472786640 1 -- 172.21.15.6:0/740872600 >> v1:172.21.15.6:6790/0 conn(0x7f946c0a4970 legacy=0x7f946c1d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f9471784640 1 -- 172.21.15.6:0/740872600 >> v1:172.21.15.6:6789/0 conn(0x7f946c0a3e80 legacy=0x7f946c1d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091151910 0 0) 0x7f946c13ee10 con 0x7f946c0a4970 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94627fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94627fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94627fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94627fc640 10 monclient(hunting): my global_id is 4343 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f9471f85640 1 -- 172.21.15.6:0/740872600 >> v1:172.21.15.203:6789/0 conn(0x7f946c09bab0 legacy=0x7f946c139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.576+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9450003530 con 0x7f946c0a4970 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2210597569 0 0) 0x7f946c0a5c80 con 0x7f946c0a3e80 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): my global_id is 4362 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f946c13ee10 con 0x7f946c0a3e80 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2653905830 0 0) 0x7f9450003530 con 0x7f946c0a4970 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f946c0a5c80 con 0x7f946c0a4970 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9464001be0 con 0x7f946c0a4970 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): dump: 2024-04-25T21:40:10.589 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2051845766 0 0) 0x7f946c13ee10 con 0x7f946c0a3e80 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9450003530 con 0x7f946c0a3e80 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f945c001c90 con 0x7f946c0a3e80 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient(hunting): dump: 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 241354976 0 0) 0x7f9450003530 con 0x7f946c0a3e80 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 >> v1:172.21.15.6:6790/0 conn(0x7f946c0a4970 legacy=0x7f946c1d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 >> v1:172.21.15.203:6789/0 conn(0x7f946c09bab0 legacy=0x7f946c139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 monclient: found mon.a 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c120c60 con 0x7f946c0a3e80 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 5 monclient: authenticate success, global_id 4362 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 10 monclient: _renew_subs 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f946c139a10 con 0x7f946c0a3e80 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 10 monclient: _renew_subs 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f946c13fd60 con 0x7f946c0a3e80 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f945c0038d0 con 0x7f946c0a3e80 2024-04-25T21:40:10.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f945c003c20 con 0x7f946c0a3e80 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 10 monclient: dump: 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.593 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f9463fff640 4 set_mon_vals no callback set 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f945c003e40 con 0x7f946c0a3e80 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f945c044220 con 0x7f946c0a3e80 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946c062e50 con 0x7f946c0a3e80 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.580+0000 7f9471f85640 10 monclient: get_auth_request con 0x7f9450036a60 auth_method 0 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.584+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f945c015540 con 0x7f946c0a3e80 2024-04-25T21:40:10.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.584+0000 7f94627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.584+0000 7f94627fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:10.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.740+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f946c1c8a90 con 0x7f9450036a60 2024-04-25T21:40:10.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94627fc640 1 -- 172.21.15.6:0/740872600 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33794 (unknown 2965378022 0 2743990926) 0x7f94680020a0 con 0x7f9450036a60 2024-04-25T21:40:10.747 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:10.747 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:10.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 >> v1:172.21.15.203:6812/3164820986 conn(0x7f9450036a60 legacy=0x7f9450038f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 10 monclient: shutdown 2024-04-25T21:40:10.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 >> v1:172.21.15.6:6789/0 conn(0x7f946c0a3e80 legacy=0x7f946c1d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 shutdown_connections 2024-04-25T21:40:10.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 >> 172.21.15.6:0/740872600 conn(0x7f946c0a5f80 msgr2=0x7f946c09ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 shutdown_connections 2024-04-25T21:40:10.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.744+0000 7f94741e9640 1 -- 172.21.15.6:0/740872600 wait complete. 2024-04-25T21:40:10.763 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-25T21:40:09.517876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164140,"kb_used_data":1992,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066900,"statfs":{"total":579820584960,"available":579652505600,"internally_reserved":0,"allocated":2039808,"data_stored":1355822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001700"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118941+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:08.118941+0000","last_peered":"2024-04-25T21:40:08.118941+0000","last_clean":"2024-04-25T21:40:08.118941+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:08.118941+0000","last_undegraded":"2024-04-25T21:40:08.118941+0000","last_fullsized":"2024-04-25T21:40:08.118941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122641+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:08.122641+0000","last_peered":"2024-04-25T21:40:08.122641+0000","last_clean":"2024-04-25T21:40:08.122641+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:08.122641+0000","last_undegraded":"2024-04-25T21:40:08.122641+0000","last_fullsized":"2024-04-25T21:40:08.122641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118903+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:08.118903+0000","last_peered":"2024-04-25T21:40:08.118903+0000","last_clean":"2024-04-25T21:40:08.118903+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:08.118903+0000","last_undegraded":"2024-04-25T21:40:08.118903+0000","last_fullsized":"2024-04-25T21:40:08.118903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122575+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:08.122575+0000","last_peered":"2024-04-25T21:40:08.122575+0000","last_clean":"2024-04-25T21:40:08.122575+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:08.122575+0000","last_undegraded":"2024-04-25T21:40:08.122575+0000","last_fullsized":"2024-04-25T21:40:08.122575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500532+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:08.500532+0000","last_peered":"2024-04-25T21:40:08.500532+0000","last_clean":"2024-04-25T21:40:08.500532+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:08.500532+0000","last_undegraded":"2024-04-25T21:40:08.500532+0000","last_fullsized":"2024-04-25T21:40:08.500532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.120646+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:08.120646+0000","last_peered":"2024-04-25T21:40:08.120646+0000","last_clean":"2024-04-25T21:40:08.120646+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:08.120646+0000","last_undegraded":"2024-04-25T21:40:08.120646+0000","last_fullsized":"2024-04-25T21:40:08.120646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118915+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:08.118915+0000","last_peered":"2024-04-25T21:40:08.118915+0000","last_clean":"2024-04-25T21:40:08.118915+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:08.118915+0000","last_undegraded":"2024-04-25T21:40:08.118915+0000","last_fullsized":"2024-04-25T21:40:08.118915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500564+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:08.500564+0000","last_peered":"2024-04-25T21:40:08.500564+0000","last_clean":"2024-04-25T21:40:08.500564+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:08.500564+0000","last_undegraded":"2024-04-25T21:40:08.500564+0000","last_fullsized":"2024-04-25T21:40:08.500564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118784+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:08.118784+0000","last_peered":"2024-04-25T21:40:08.118784+0000","last_clean":"2024-04-25T21:40:08.118784+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:08.118784+0000","last_undegraded":"2024-04-25T21:40:08.118784+0000","last_fullsized":"2024-04-25T21:40:08.118784+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":579263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":575219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:10.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient: get_monmap_and_config 2024-04-25T21:40:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient: monmap: 2024-04-25T21:40:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:10.873267+0000 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:10.873267+0000 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:10.872 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 1 Processor -- start 2024-04-25T21:40:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 1 -- start start 2024-04-25T21:40:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient: init 2024-04-25T21:40:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient(hunting): picked mon.noname-a con 0x7f252c09e860 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient(hunting): picked mon.noname-b con 0x7f252c0a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient(hunting): picked mon.noname-c con 0x7f252c0a1680 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c09c130 con 0x7f252c09e860 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c099520 con 0x7f252c0a1680 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c099b10 con 0x7f252c0a1270 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2531157640 10 monclient(hunting): authenticate will time out at 969.783037s 2024-04-25T21:40:10.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2523fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f252c0a1270 0x7f252c139b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45860/0 (socket says 172.21.15.6:45860) 2024-04-25T21:40:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f252ad9d640 1 --1- 172.21.15.6:0/3998147094 >> v1:172.21.15.6:6789/0 conn(0x7f252c09e860 0x7f252c0a0b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47822/0 (socket says 172.21.15.6:47822) 2024-04-25T21:40:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.868+0000 7f2523fff640 1 -- 172.21.15.6:0/3998147094 learned_addr learned my addr 172.21.15.6:0/3998147094 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252ad9d640 1 -- 172.21.15.6:0/3998147094 >> v1:172.21.15.6:6789/0 conn(0x7f252c09e860 legacy=0x7f252c0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2523fff640 1 -- 172.21.15.6:0/3998147094 >> v1:172.21.15.6:6790/0 conn(0x7f252c0a1270 legacy=0x7f252c139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538230820 0 0) 0x7f251c001460 con 0x7f252c09e860 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): my global_id is 4365 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2518003530 con 0x7f252c09e860 2024-04-25T21:40:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2342606696 0 0) 0x7f252c099b10 con 0x7f252c0a1270 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient(hunting): my global_id is 4346 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f251c001460 con 0x7f252c0a1270 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252b59e640 1 -- 172.21.15.6:0/3998147094 >> v1:172.21.15.203:6789/0 conn(0x7f252c0a1680 legacy=0x7f252c13d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3403781670 0 0) 0x7f251c001460 con 0x7f252c0a1270 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 >> v1:172.21.15.203:6789/0 conn(0x7f252c0a1680 legacy=0x7f252c13d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 >> v1:172.21.15.6:6789/0 conn(0x7f252c09e860 legacy=0x7f252c0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 monclient: found mon.noname-b 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c09a0d0 con 0x7f252c0a1270 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f25140018d0 con 0x7f252c0a1270 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 5 monclient: authenticate success, global_id 4346 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: dump: 2024-04-25T21:40:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2523fff640 1 --1- 172.21.15.6:0/3998147094 >> v1:172.21.15.6:6790/0 conn(0x7f252c0a1270 0x7f252c139b10 :-1 s=OPENED pgs=115 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f2514004a50 mon_map magic: 0 2024-04-25T21:40:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2514001c20 con 0x7f252c0a1270 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 4 set_mon_vals no callback set 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient: shutdown 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- 172.21.15.6:0/3998147094 >> v1:172.21.15.6:6790/0 conn(0x7f252c0a1270 legacy=0x7f252c139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 1 -- 172.21.15.6:0/3998147094 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2514004a50 con 0x7f252c0a1270 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252a59c640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- 172.21.15.6:0/3998147094 shutdown_connections 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- 172.21.15.6:0/3998147094 >> 172.21.15.6:0/3998147094 conn(0x7f252c0a1c50 msgr2=0x7f252c0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- 172.21.15.6:0/3998147094 shutdown_connections 2024-04-25T21:40:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- 172.21.15.6:0/3998147094 wait complete. 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient: build_initial_monmap 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient: monmap: 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 Processor -- start 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- start start 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient: init 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient(hunting): picked mon.noname-b con 0x7f252c09e860 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient(hunting): picked mon.noname-a con 0x7f252c0a1270 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient(hunting): picked mon.noname-c con 0x7f252c0a1680 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c08ce20 con 0x7f252c0a1270 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c072b40 con 0x7f252c09e860 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252c08d0e0 con 0x7f252c0a1680 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2531157640 10 monclient(hunting): authenticate will time out at 969.787850s 2024-04-25T21:40:10.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2523fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f252c0a1270 0x7f252c1ced90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47824/0 (socket says 172.21.15.6:47824) 2024-04-25T21:40:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f2523fff640 1 -- 172.21.15.6:0/1789599615 learned_addr learned my addr 172.21.15.6:0/1789599615 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.872+0000 7f252b59e640 1 --1- 172.21.15.6:0/1789599615 >> v1:172.21.15.6:6790/0 conn(0x7f252c0a1680 0x7f252c1d24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45862/0 (socket says 172.21.15.6:45862) 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2523fff640 1 -- 172.21.15.6:0/1789599615 >> v1:172.21.15.6:6789/0 conn(0x7f252c0a1270 legacy=0x7f252c1ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f252b59e640 1 -- 172.21.15.6:0/1789599615 >> v1:172.21.15.6:6790/0 conn(0x7f252c0a1680 legacy=0x7f252c1d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 904394450 0 0) 0x7f252c08ce20 con 0x7f252c0a1270 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): my global_id is 4368 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f250c003530 con 0x7f252c0a1270 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2864019614 0 0) 0x7f252c08d0e0 con 0x7f252c0a1680 2024-04-25T21:40:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): my global_id is 4349 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f252c08ce20 con 0x7f252c0a1680 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f252ad9d640 1 -- 172.21.15.6:0/1789599615 >> v1:172.21.15.203:6789/0 conn(0x7f252c09e860 legacy=0x7f252c13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4276489548 0 0) 0x7f252c08ce20 con 0x7f252c0a1680 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252c08d0e0 con 0x7f252c0a1680 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2524001e80 con 0x7f252c0a1680 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): dump: 2024-04-25T21:40:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1232328602 0 0) 0x7f250c003530 con 0x7f252c0a1270 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f252c08ce20 con 0x7f252c0a1270 2024-04-25T21:40:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2514004990 con 0x7f252c0a1270 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient(hunting): dump: 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.892 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3540357480 0 0) 0x7f252c08d0e0 con 0x7f252c0a1680 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 >> v1:172.21.15.203:6789/0 conn(0x7f252c09e860 legacy=0x7f252c13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 >> v1:172.21.15.6:6789/0 conn(0x7f252c0a1270 legacy=0x7f252c1ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 monclient: found mon.c 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c08d3c0 con 0x7f252c0a1680 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:10.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 5 monclient: authenticate success, global_id 4349 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 10 monclient: _renew_subs 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f252c097990 con 0x7f252c0a1680 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 10 monclient: _renew_subs 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f252c092090 con 0x7f252c0a1680 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2524003c50 con 0x7f252c0a1680 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2524003fa0 con 0x7f252c0a1680 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2529d9b640 4 set_mon_vals no callback set 2024-04-25T21:40:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 10 monclient: dump: 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2510005090 con 0x7f252c0a1680 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.876+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f2524004180 con 0x7f252c0a1680 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.880+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f2524044650 con 0x7f252c0a1680 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.880+0000 7f252ad9d640 10 monclient: get_auth_request con 0x7f250c036a80 auth_method 0 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.884+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f25240159f0 con 0x7f252c0a1680 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.884+0000 7f25237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:10.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:10.884+0000 7f25237fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:11.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.036+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2510000ba0 con 0x7f250c036a80 2024-04-25T21:40:11.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.040+0000 7f25237fe640 1 -- 172.21.15.6:0/1789599615 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33794 (unknown 2965378022 0 2743990926) 0x7f2510000ba0 con 0x7f250c036a80 2024-04-25T21:40:11.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:11.044 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 >> v1:172.21.15.203:6812/3164820986 conn(0x7f250c036a80 legacy=0x7f250c038f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 10 monclient: shutdown 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 >> v1:172.21.15.6:6790/0 conn(0x7f252c0a1680 legacy=0x7f252c1d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 shutdown_connections 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 >> 172.21.15.6:0/1789599615 conn(0x7f252c0a1c50 msgr2=0x7f252c0987d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 shutdown_connections 2024-04-25T21:40:11.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.044+0000 7f2531157640 1 -- 172.21.15.6:0/1789599615 wait complete. 2024-04-25T21:40:11.061 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-25T21:40:09.517876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164140,"kb_used_data":1992,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066900,"statfs":{"total":579820584960,"available":579652505600,"internally_reserved":0,"allocated":2039808,"data_stored":1355822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001700"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118941+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:08.118941+0000","last_peered":"2024-04-25T21:40:08.118941+0000","last_clean":"2024-04-25T21:40:08.118941+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:08.118941+0000","last_undegraded":"2024-04-25T21:40:08.118941+0000","last_fullsized":"2024-04-25T21:40:08.118941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122641+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:08.122641+0000","last_peered":"2024-04-25T21:40:08.122641+0000","last_clean":"2024-04-25T21:40:08.122641+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:08.122641+0000","last_undegraded":"2024-04-25T21:40:08.122641+0000","last_fullsized":"2024-04-25T21:40:08.122641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118903+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:08.118903+0000","last_peered":"2024-04-25T21:40:08.118903+0000","last_clean":"2024-04-25T21:40:08.118903+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:08.118903+0000","last_undegraded":"2024-04-25T21:40:08.118903+0000","last_fullsized":"2024-04-25T21:40:08.118903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122575+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:08.122575+0000","last_peered":"2024-04-25T21:40:08.122575+0000","last_clean":"2024-04-25T21:40:08.122575+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:08.122575+0000","last_undegraded":"2024-04-25T21:40:08.122575+0000","last_fullsized":"2024-04-25T21:40:08.122575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500532+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:08.500532+0000","last_peered":"2024-04-25T21:40:08.500532+0000","last_clean":"2024-04-25T21:40:08.500532+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:08.500532+0000","last_undegraded":"2024-04-25T21:40:08.500532+0000","last_fullsized":"2024-04-25T21:40:08.500532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.120646+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:08.120646+0000","last_peered":"2024-04-25T21:40:08.120646+0000","last_clean":"2024-04-25T21:40:08.120646+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:08.120646+0000","last_undegraded":"2024-04-25T21:40:08.120646+0000","last_fullsized":"2024-04-25T21:40:08.120646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118915+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:08.118915+0000","last_peered":"2024-04-25T21:40:08.118915+0000","last_clean":"2024-04-25T21:40:08.118915+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:08.118915+0000","last_undegraded":"2024-04-25T21:40:08.118915+0000","last_fullsized":"2024-04-25T21:40:08.118915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500564+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:08.500564+0000","last_peered":"2024-04-25T21:40:08.500564+0000","last_clean":"2024-04-25T21:40:08.500564+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:08.500564+0000","last_undegraded":"2024-04-25T21:40:08.500564+0000","last_fullsized":"2024-04-25T21:40:08.500564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118784+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:08.118784+0000","last_peered":"2024-04-25T21:40:08.118784+0000","last_clean":"2024-04-25T21:40:08.118784+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:08.118784+0000","last_undegraded":"2024-04-25T21:40:08.118784+0000","last_fullsized":"2024-04-25T21:40:08.118784+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607555,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":579263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":575219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:11.061 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T21:40:11.062 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T21:40:11.062 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T21:40:11.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient: get_monmap_and_config 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient: build_initial_monmap 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient: monmap: 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:11.184119+0000 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:11.184119+0000 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.182 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:11.183 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:11.183 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:11.183 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 1 Processor -- start 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 1 -- start start 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient: init 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient(hunting): picked mon.noname-a con 0x7fa61c0a3e80 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient(hunting): picked mon.noname-c con 0x7fa61c0a4970 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient(hunting): picked mon.noname-b con 0x7fa61c09bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:11.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c09c590 con 0x7fa61c0a3e80 2024-04-25T21:40:11.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c09adf0 con 0x7fa61c0a4970 2024-04-25T21:40:11.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c09bf20 con 0x7fa61c09bab0 2024-04-25T21:40:11.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:11.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa620dad640 10 monclient(hunting): authenticate will time out at 970.093917s 2024-04-25T21:40:11.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa61a59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa61c0a3e80 0x7fa61c0a4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47826/0 (socket says 172.21.15.6:47826) 2024-04-25T21:40:11.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa61a59c640 1 -- 172.21.15.6:0/748636150 learned_addr learned my addr 172.21.15.6:0/748636150 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:11.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.180+0000 7fa61ad9d640 1 --1- 172.21.15.6:0/748636150 >> v1:172.21.15.6:6790/0 conn(0x7fa61c09bab0 0x7fa61c13f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45868/0 (socket says 172.21.15.6:45868) 2024-04-25T21:40:11.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61ad9d640 1 -- 172.21.15.6:0/748636150 >> v1:172.21.15.6:6790/0 conn(0x7fa61c09bab0 legacy=0x7fa61c13f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T21:40:11.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61ad9d640 1 -- 172.21.15.6:0/748636150 >> v1:172.21.15.6:6790/0 conn(0x7fa61c09bab0 legacy=0x7fa61c13f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T21:40:11.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61ad9d640 1 --1- 172.21.15.6:0/748636150 >> v1:172.21.15.6:6790/0 conn(0x7fa61c09bab0 0x7fa61c13f7c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61a59c640 1 -- 172.21.15.6:0/748636150 >> v1:172.21.15.6:6789/0 conn(0x7fa61c0a3e80 legacy=0x7fa61c0a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3228914399 0 0) 0x7fa61c09c590 con 0x7fa61c0a3e80 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient(hunting): my global_id is 4371 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa608003530 con 0x7fa61c0a3e80 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa619d9b640 1 -- 172.21.15.6:0/748636150 >> v1:172.21.15.203:6789/0 conn(0x7fa61c0a4970 legacy=0x7fa61c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 215477356 0 0) 0x7fa608003530 con 0x7fa61c0a3e80 2024-04-25T21:40:11.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 >> v1:172.21.15.6:6790/0 conn(0x7fa61c09bab0 legacy=0x7fa61c13f7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 >> v1:172.21.15.203:6789/0 conn(0x7fa61c0a4970 legacy=0x7fa61c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: _finish_hunting 0 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 monclient: found mon.noname-a 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61c06eaf0 con 0x7fa61c0a3e80 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: _finish_auth 0 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: _check_auth_tickets 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa6140018d0 con 0x7fa61c0a3e80 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: dump: 2024-04-25T21:40:11.189 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 5 monclient: authenticate success, global_id 4371 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa614001c20 con 0x7fa61c0a3e80 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:11.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 1 -- 172.21.15.6:0/748636150 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa614004a50 con 0x7fa61c0a3e80 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61959a640 10 monclient: dump: 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:11.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 4 set_mon_vals no callback set 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient: shutdown 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- 172.21.15.6:0/748636150 >> v1:172.21.15.6:6789/0 conn(0x7fa61c0a3e80 legacy=0x7fa61c0a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- 172.21.15.6:0/748636150 shutdown_connections 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- 172.21.15.6:0/748636150 >> 172.21.15.6:0/748636150 conn(0x7fa61c0a5f80 msgr2=0x7fa61c0a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- 172.21.15.6:0/748636150 shutdown_connections 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- 172.21.15.6:0/748636150 wait complete. 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient: build_initial_monmap 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient: monmap: 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:11.193 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:11.193 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:11.193 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 Processor -- start 2024-04-25T21:40:11.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- start start 2024-04-25T21:40:11.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient: init 2024-04-25T21:40:11.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:11.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:11.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient(hunting): picked mon.noname-b con 0x7fa61c09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient(hunting): picked mon.noname-c con 0x7fa61c0a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient(hunting): picked mon.noname-a con 0x7fa61c0a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c120cd0 con 0x7fa61c0a4970 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c0a5c80 con 0x7fa61c09bab0 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa61c139c70 con 0x7fa61c0a3e80 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa620dad640 10 monclient(hunting): authenticate will time out at 970.098875s 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61ad9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa61c0a4970 0x7fa61c1d40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47842/0 (socket says 172.21.15.6:47842) 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.184+0000 7fa61ad9d640 1 -- 172.21.15.6:0/4281050304 learned_addr learned my addr 172.21.15.6:0/4281050304 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa619d9b640 1 -- 172.21.15.6:0/4281050304 >> v1:172.21.15.6:6790/0 conn(0x7fa61c0a3e80 legacy=0x7fa61c1d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa61ad9d640 1 -- 172.21.15.6:0/4281050304 >> v1:172.21.15.6:6789/0 conn(0x7fa61c0a4970 legacy=0x7fa61c1d40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1688317514 0 0) 0x7fa61c139c70 con 0x7fa61c0a3e80 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): my global_id is 4352 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5fc003530 con 0x7fa61c0a3e80 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556056199 0 0) 0x7fa61c120cd0 con 0x7fa61c0a4970 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): my global_id is 4374 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa61c139c70 con 0x7fa61c0a4970 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa61a59c640 1 -- 172.21.15.6:0/4281050304 >> v1:172.21.15.203:6789/0 conn(0x7fa61c09bab0 legacy=0x7fa61c1cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 925527561 0 0) 0x7fa5fc003530 con 0x7fa61c0a3e80 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa61c120cd0 con 0x7fa61c0a3e80 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa604001e80 con 0x7fa61c0a3e80 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:11.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.199 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1528931720 0 0) 0x7fa61c139c70 con 0x7fa61c0a4970 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5fc003530 con 0x7fa61c0a4970 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa60c001420 con 0x7fa61c0a4970 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.200 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1913371407 0 0) 0x7fa61c120cd0 con 0x7fa61c0a3e80 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 >> v1:172.21.15.203:6789/0 conn(0x7fa61c09bab0 legacy=0x7fa61c1cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 >> v1:172.21.15.6:6789/0 conn(0x7fa61c0a4970 legacy=0x7fa61c1d40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 monclient: found mon.c 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61c13ee60 con 0x7fa61c0a3e80 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 5 monclient: authenticate success, global_id 4352 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 10 monclient: _renew_subs 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa61c1209f0 con 0x7fa61c0a3e80 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 10 monclient: _renew_subs 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa61c139990 con 0x7fa61c0a3e80 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa604003c50 con 0x7fa61c0a3e80 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa604003fa0 con 0x7fa61c0a3e80 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 10 monclient: dump: 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa618d99640 4 set_mon_vals no callback set 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa600005090 con 0x7fa61c0a3e80 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fa604004140 con 0x7fa61c0a3e80 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7fa604044710 con 0x7fa61c0a3e80 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.188+0000 7fa61a59c640 10 monclient: get_auth_request con 0x7fa5fc036af0 auth_method 0 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.192+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fa604015b30 con 0x7fa61c0a3e80 2024-04-25T21:40:11.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.192+0000 7fa612ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.192+0000 7fa612ffd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.380+0000 7fa620dad640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-25T21:40:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.380+0000 7fa620dad640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-25T21:40:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.380+0000 7fa620dad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.380+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa600005380 con 0x7fa61c0a3e80 2024-04-25T21:40:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.380+0000 7fa612ffd640 1 -- 172.21.15.6:0/4281050304 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fa604015430 con 0x7fa61c0a3e80 2024-04-25T21:40:11.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.380+0000 7fa612ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-25T21:40:11.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.380+0000 7fa612ffd640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:11.385 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T21:40:11.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 >> v1:172.21.15.203:6812/3164820986 conn(0x7fa5fc036af0 legacy=0x7fa5fc038fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 10 monclient: shutdown 2024-04-25T21:40:11.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:11.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 >> v1:172.21.15.6:6790/0 conn(0x7fa61c0a3e80 legacy=0x7fa61c1d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 shutdown_connections 2024-04-25T21:40:11.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 >> 172.21.15.6:0/4281050304 conn(0x7fa61c0a5f80 msgr2=0x7fa61c09f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:11.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 shutdown_connections 2024-04-25T21:40:11.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.384+0000 7fa620dad640 1 -- 172.21.15.6:0/4281050304 wait complete. 2024-04-25T21:40:11.401 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T21:40:11.402 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-04-25T21:40:11.415 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-04-25T21:40:11.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.520+0000 7f7413bc4640 10 monclient: get_monmap_and_config 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.520+0000 7f7413bc4640 10 monclient: build_initial_monmap 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.520+0000 7f7413bc4640 10 monclient: monmap: 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:11.527061+0000 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:11.527061+0000 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:11.526 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 1 Processor -- start 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 1 -- start start 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient: init 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient(hunting): picked mon.noname-a con 0x7f740c09df10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient(hunting): picked mon.noname-c con 0x7f740c09cb10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient(hunting): picked mon.noname-b con 0x7f740c1399f0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f740c09bc10 con 0x7f740c09df10 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f740c073230 con 0x7f740c09cb10 2024-04-25T21:40:11.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f740c09f2f0 con 0x7f740c1399f0 2024-04-25T21:40:11.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:11.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 10 monclient(hunting): authenticate will time out at 970.436913s 2024-04-25T21:40:11.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7412161640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f740c1399f0 0x7f740c13bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45888/0 (socket says 172.21.15.6:45888) 2024-04-25T21:40:11.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7411960640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f740c09df10 0x7f740c09e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47858/0 (socket says 172.21.15.6:47858) 2024-04-25T21:40:11.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7412161640 1 -- 172.21.15.6:0/3417478883 learned_addr learned my addr 172.21.15.6:0/3417478883 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:11.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7411960640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6789/0 conn(0x7f740c09df10 legacy=0x7f740c09e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7412161640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6790/0 conn(0x7f740c1399f0 legacy=0x7f740c13bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031405592 0 0) 0x7f740c09bc10 con 0x7f740c09df10 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): my global_id is 4377 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f8003530 con 0x7f740c09df10 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741115f640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.203:6789/0 conn(0x7f740c09cb10 legacy=0x7f740c1382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2958060303 0 0) 0x7f740c09f2f0 con 0x7f740c1399f0 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): my global_id is 4355 2024-04-25T21:40:11.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f740c09bc10 con 0x7f740c1399f0 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7412161640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6790/0 conn(0x7f740c1399f0 legacy=0x7f740c13bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7412161640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6790/0 conn(0x7f740c1399f0 legacy=0x7f740c13bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7412161640 1 --1- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6790/0 conn(0x7f740c1399f0 0x7f740c13bde0 :-1 s=OPENED pgs=119 cs=1 l=1).handle_message read tag failed 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7412161640 1 --1- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6790/0 conn(0x7f740c1399f0 0x7f740c13bde0 :-1 s=OPENED pgs=119 cs=1 l=1).fault on lossy channel, failing 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6790/0 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3747739417 0 0) 0x7f73f8003530 con 0x7f740c09df10 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6790/0 conn(0x7f740c1399f0 legacy=0x7f740c13bde0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.203:6789/0 conn(0x7f740c09cb10 legacy=0x7f740c1382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient: _finish_hunting 0 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 monclient: found mon.noname-a 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740c09b930 con 0x7f740c09df10 2024-04-25T21:40:11.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient: _finish_auth 0 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient: _check_auth_tickets 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f74080018d0 con 0x7f740c09df10 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f741095e640 10 monclient: dump: 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 5 monclient: authenticate success, global_id 4377 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.524+0000 7f7413bc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7408001c20 con 0x7f740c09df10 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f741095e640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f741095e640 1 -- 172.21.15.6:0/3417478883 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7408004990 con 0x7f740c09df10 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 4 set_mon_vals no callback set 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f741095e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f741095e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f741095e640 10 monclient: dump: 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.534 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient: shutdown 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- 172.21.15.6:0/3417478883 >> v1:172.21.15.6:6789/0 conn(0x7f740c09df10 legacy=0x7f740c09e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- 172.21.15.6:0/3417478883 shutdown_connections 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- 172.21.15.6:0/3417478883 >> 172.21.15.6:0/3417478883 conn(0x7f740c0a1c50 msgr2=0x7f740c0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- 172.21.15.6:0/3417478883 shutdown_connections 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- 172.21.15.6:0/3417478883 wait complete. 2024-04-25T21:40:11.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient: build_initial_monmap 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient: monmap: 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 Processor -- start 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- start start 2024-04-25T21:40:11.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient: init 2024-04-25T21:40:11.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:11.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:11.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient(hunting): picked mon.noname-a con 0x7f740c09df10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:11.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient(hunting): picked mon.noname-b con 0x7f740c1399f0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient(hunting): picked mon.noname-c con 0x7f740c070950 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f740c09ed00 con 0x7f740c09df10 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f740c0701c0 con 0x7f740c1399f0 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f740c139180 con 0x7f740c070950 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7413bc4640 10 monclient(hunting): authenticate will time out at 970.442050s 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7411960640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f740c09df10 0x7f740c138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47874/0 (socket says 172.21.15.6:47874) 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.528+0000 7f7411960640 1 -- 172.21.15.6:0/4091950108 learned_addr learned my addr 172.21.15.6:0/4091950108 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7411960640 1 -- 172.21.15.6:0/4091950108 >> v1:172.21.15.6:6789/0 conn(0x7f740c09df10 legacy=0x7f740c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7412161640 1 -- 172.21.15.6:0/4091950108 >> v1:172.21.15.6:6790/0 conn(0x7f740c070950 legacy=0x7f740c1cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123079266 0 0) 0x7f740c139180 con 0x7f740c070950 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): my global_id is 4358 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f0003530 con 0x7f740c070950 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2545245993 0 0) 0x7f740c09b930 con 0x7f740c09df10 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): my global_id is 4380 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f740c139180 con 0x7f740c09df10 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f741115f640 1 -- 172.21.15.6:0/4091950108 >> v1:172.21.15.203:6789/0 conn(0x7f740c1399f0 legacy=0x7f740c138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1478089614 0 0) 0x7f73f0003530 con 0x7f740c070950 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f740c09b930 con 0x7f740c070950 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7404001f40 con 0x7f740c070950 2024-04-25T21:40:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): dump: 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4223396677 0 0) 0x7f740c139180 con 0x7f740c09df10 2024-04-25T21:40:11.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73f0003530 con 0x7f740c09df10 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7408004990 con 0x7f740c09df10 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient(hunting): dump: 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.544 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4071248026 0 0) 0x7f740c09b930 con 0x7f740c070950 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 >> v1:172.21.15.203:6789/0 conn(0x7f740c1399f0 legacy=0x7f740c138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 >> v1:172.21.15.6:6789/0 conn(0x7f740c09df10 legacy=0x7f740c138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 monclient: found mon.c 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740c06e650 con 0x7f740c070950 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 5 monclient: authenticate success, global_id 4358 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 10 monclient: _renew_subs 2024-04-25T21:40:11.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f740c06fee0 con 0x7f740c070950 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 10 monclient: _renew_subs 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f740c1d1560 con 0x7f740c070950 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7404004e20 con 0x7f740c070950 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7404005170 con 0x7f740c070950 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 10 monclient: dump: 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.546 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7403fff640 4 set_mon_vals no callback set 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f7404005390 con 0x7f740c070950 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f4005090 con 0x7f740c070950 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f7404044820 con 0x7f740c070950 2024-04-25T21:40:11.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.532+0000 7f7411960640 10 monclient: get_auth_request con 0x7f73f0036b10 auth_method 0 2024-04-25T21:40:11.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.536+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f7404015bc0 con 0x7f740c070950 2024-04-25T21:40:11.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.536+0000 7f74027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.536+0000 7f74027fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.692+0000 7f7413bc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.692+0000 7f7413bc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.692+0000 7f7413bc4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.692+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73f4005380 con 0x7f740c070950 2024-04-25T21:40:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.692+0000 7f74027fc640 1 -- 172.21.15.6:0/4091950108 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10773 (unknown 941590788 0 124421770) 0x7f74040154c0 con 0x7f740c070950 2024-04-25T21:40:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.692+0000 7f74027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.692+0000 7f74027fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:11.697 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:11.697 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:40:08.112324+0000","last_up_change":"2024-04-25T21:39:58.313547+0000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T21:39:59.601307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T21:40:05.399371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":763650169}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":763650169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":763650169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":763650169}]},"public_addr":"172.21.15.6:6804/763650169","cluster_addr":"172.21.15.6:6805/763650169","heartbeat_back_addr":"172.21.15.6:6809/763650169","heartbeat_front_addr":"172.21.15.6:6807/763650169","state":["exists","up"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":2126016572}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":2126016572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":2126016572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":2126016572}]},"public_addr":"172.21.15.6:6806/2126016572","cluster_addr":"172.21.15.6:6808/2126016572","heartbeat_back_addr":"172.21.15.6:6811/2126016572","heartbeat_front_addr":"172.21.15.6:6810/2126016572","state":["exists","up"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":3408092622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":3408092622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":3408092622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":3408092622}]},"public_addr":"172.21.15.6:6800/3408092622","cluster_addr":"172.21.15.6:6801/3408092622","heartbeat_back_addr":"172.21.15.6:6803/3408092622","heartbeat_front_addr":"172.21.15.6:6802/3408092622","state":["exists","up"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3201437079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3201437079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":3201437079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":3201437079}]},"public_addr":"172.21.15.203:6800/3201437079","cluster_addr":"172.21.15.203:6802/3201437079","heartbeat_back_addr":"172.21.15.203:6806/3201437079","heartbeat_front_addr":"172.21.15.203:6804/3201437079","state":["exists","up"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":1405305648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":1405305648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1405305648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1405305648}]},"public_addr":"172.21.15.203:6801/1405305648","cluster_addr":"172.21.15.203:6803/1405305648","heartbeat_back_addr":"172.21.15.203:6807/1405305648","heartbeat_front_addr":"172.21.15.203:6805/1405305648","state":["exists","up"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1687281622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1687281622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1687281622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1687281622}]},"public_addr":"172.21.15.203:6808/1687281622","cluster_addr":"172.21.15.203:6809/1687281622","heartbeat_back_addr":"172.21.15.203:6811/1687281622","heartbeat_front_addr":"172.21.15.203:6810/1687281622","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.481868+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.631515+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.371692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.722323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.712123+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.636554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 >> v1:172.21.15.203:6812/3164820986 conn(0x7f73f0036b10 legacy=0x7f73f0038fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 10 monclient: shutdown 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 >> v1:172.21.15.6:6790/0 conn(0x7f740c070950 legacy=0x7f740c1cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 shutdown_connections 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 >> 172.21.15.6:0/4091950108 conn(0x7f740c0a1c50 msgr2=0x7f740c0a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 shutdown_connections 2024-04-25T21:40:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.696+0000 7f7413bc4640 1 -- 172.21.15.6:0/4091950108 wait complete. 2024-04-25T21:40:11.714 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T21:39:59.601307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-25T21:40:05.399371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T21:40:11.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.828+0000 7f7538037640 10 monclient: get_monmap_and_config 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.828+0000 7f7538037640 10 monclient: build_initial_monmap 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.828+0000 7f7538037640 10 monclient: monmap: 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:11.835135+0000 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:11.835135+0000 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:11.833 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:11.834 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:11.834 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 1 Processor -- start 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 1 -- start start 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient: init 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient(hunting): picked mon.noname-a con 0x7f75300a3e80 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient(hunting): picked mon.noname-c con 0x7f75300a4970 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient(hunting): picked mon.noname-b con 0x7f753013d5a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f753009adf0 con 0x7f75300a3e80 2024-04-25T21:40:11.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f753009b340 con 0x7f75300a4970 2024-04-25T21:40:11.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f753009b620 con 0x7f753013d5a0 2024-04-25T21:40:11.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:11.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient(hunting): authenticate will time out at 970.744918s 2024-04-25T21:40:11.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7535dd3640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f75300a3e80 0x7f75300a4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47882/0 (socket says 172.21.15.6:47882) 2024-04-25T21:40:11.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7535dd3640 1 -- 172.21.15.6:0/840592724 learned_addr learned my addr 172.21.15.6:0/840592724 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7535dd3640 1 -- 172.21.15.6:0/840592724 >> v1:172.21.15.6:6789/0 conn(0x7f75300a3e80 legacy=0x7f75300a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f75365d4640 1 -- 172.21.15.6:0/840592724 >> v1:172.21.15.6:6790/0 conn(0x7f753013d5a0 legacy=0x7f753013f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035398026 0 0) 0x7f753009adf0 con 0x7f75300a3e80 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): my global_id is 4383 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f751c003530 con 0x7f75300a3e80 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142448844 0 0) 0x7f753009b620 con 0x7f753013d5a0 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient(hunting): my global_id is 4361 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f753009adf0 con 0x7f753013d5a0 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1109758639 0 0) 0x7f751c003530 con 0x7f75300a3e80 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 >> v1:172.21.15.6:6790/0 conn(0x7f753013d5a0 legacy=0x7f753013f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 >> v1:172.21.15.203:6789/0 conn(0x7f75300a4970 legacy=0x7f753013be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: _finish_hunting 0 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 monclient: found mon.noname-a 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753006eaf0 con 0x7f75300a3e80 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: _finish_auth 0 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: _check_auth_tickets 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f752c0018d0 con 0x7f75300a3e80 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 5 monclient: authenticate success, global_id 4383 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: dump: 2024-04-25T21:40:11.841 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f752c001c20 con 0x7f75300a3e80 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 1 -- 172.21.15.6:0/840592724 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f752c004a50 con 0x7f75300a3e80 2024-04-25T21:40:11.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7534dd1640 10 monclient: dump: 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:11.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.832+0000 7f7538037640 4 set_mon_vals no callback set 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient: shutdown 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- 172.21.15.6:0/840592724 >> v1:172.21.15.6:6789/0 conn(0x7f75300a3e80 legacy=0x7f75300a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- 172.21.15.6:0/840592724 shutdown_connections 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- 172.21.15.6:0/840592724 >> 172.21.15.6:0/840592724 conn(0x7f75300a9e30 msgr2=0x7f75300a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- 172.21.15.6:0/840592724 shutdown_connections 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- 172.21.15.6:0/840592724 wait complete. 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient: build_initial_monmap 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient: monmap: 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:11.844 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 Processor -- start 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- start start 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient: init 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient(hunting): picked mon.noname-c con 0x7f75300a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient(hunting): picked mon.noname-a con 0x7f75300a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient(hunting): picked mon.noname-b con 0x7f753013d5a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75300a8790 con 0x7f75300a4970 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f753013d2c0 con 0x7f753013d5a0 2024-04-25T21:40:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f753013a240 con 0x7f75300a3e80 2024-04-25T21:40:11.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:11.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7538037640 10 monclient(hunting): authenticate will time out at 970.749965s 2024-04-25T21:40:11.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7535dd3640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f75300a3e80 0x7f75301361c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:45926/0 (socket says 172.21.15.6:45926) 2024-04-25T21:40:11.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.836+0000 7f7535dd3640 1 -- 172.21.15.6:0/2785748696 learned_addr learned my addr 172.21.15.6:0/2785748696 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7535dd3640 1 -- 172.21.15.6:0/2785748696 >> v1:172.21.15.6:6790/0 conn(0x7f75300a3e80 legacy=0x7f75301361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75355d2640 1 -- 172.21.15.6:0/2785748696 >> v1:172.21.15.6:6789/0 conn(0x7f75300a4970 legacy=0x7f75301d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 516695982 0 0) 0x7f753013a240 con 0x7f75300a3e80 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): my global_id is 4364 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7514003560 con 0x7f75300a3e80 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421843849 0 0) 0x7f75300a8790 con 0x7f75300a4970 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): my global_id is 4386 2024-04-25T21:40:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f753013a240 con 0x7f75300a4970 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 186156564 0 0) 0x7f753013a240 con 0x7f75300a4970 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f75300a8790 con 0x7f75300a4970 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7520004200 con 0x7f75300a4970 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): dump: 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75365d4640 1 -- 172.21.15.6:0/2785748696 >> v1:172.21.15.203:6789/0 conn(0x7f753013d5a0 legacy=0x7f75301d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1757836171 0 0) 0x7f7514003560 con 0x7f75300a3e80 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f753013a240 con 0x7f75300a3e80 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f752c004990 con 0x7f75300a3e80 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient(hunting): dump: 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2377731171 0 0) 0x7f75300a8790 con 0x7f75300a4970 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 >> v1:172.21.15.6:6790/0 conn(0x7f75300a3e80 legacy=0x7f75301361c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 >> v1:172.21.15.203:6789/0 conn(0x7f753013d5a0 legacy=0x7f75301d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 monclient: found mon.a 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753013c530 con 0x7f75300a4970 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 5 monclient: authenticate success, global_id 4386 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 10 monclient: _renew_subs 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f753013ab20 con 0x7f75300a4970 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 10 monclient: _renew_subs 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7530137a60 con 0x7f75300a4970 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7520001190 con 0x7f75300a4970 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f75200014e0 con 0x7f75300a4970 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 10 monclient: dump: 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7527fff640 4 set_mon_vals no callback set 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7518005090 con 0x7f75300a4970 2024-04-25T21:40:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f7520001680 con 0x7f75300a4970 2024-04-25T21:40:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f75200443b0 con 0x7f75300a4970 2024-04-25T21:40:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.840+0000 7f7535dd3640 10 monclient: get_auth_request con 0x7f7514036b90 auth_method 0 2024-04-25T21:40:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.844+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f75200156d0 con 0x7f75300a4970 2024-04-25T21:40:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.844+0000 7f75267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.844+0000 7f75267fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:12.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.996+0000 7f7538037640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-25T21:40:12.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.996+0000 7f7538037640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-25T21:40:12.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.996+0000 7f7538037640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:11.996+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7518005b90 con 0x7f75300a4970 2024-04-25T21:40:12.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f75267fc640 1 -- 172.21.15.6:0/2785748696 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (unknown 4170632370 0 2170607528) 0x7f7520019380 con 0x7f75300a4970 2024-04-25T21:40:12.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f75267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-25T21:40:12.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f75267fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:12.002 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 >> v1:172.21.15.203:6812/3164820986 conn(0x7f7514036b90 legacy=0x7f7514039050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 10 monclient: shutdown 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 >> v1:172.21.15.6:6789/0 conn(0x7f75300a4970 legacy=0x7f75301d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 shutdown_connections 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 >> 172.21.15.6:0/2785748696 conn(0x7f75300a9e30 msgr2=0x7f75300a3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 shutdown_connections 2024-04-25T21:40:12.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.000+0000 7f7538037640 1 -- 172.21.15.6:0/2785748696 wait complete. 2024-04-25T21:40:12.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-25T21:40:12.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.124+0000 7f94f4f6d640 10 monclient: get_monmap_and_config 2024-04-25T21:40:12.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.124+0000 7f94f4f6d640 10 monclient: build_initial_monmap 2024-04-25T21:40:12.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.124+0000 7f94f4f6d640 10 monclient: monmap: 2024-04-25T21:40:12.129 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:12.129 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.129 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:12.131439+0000 2024-04-25T21:40:12.129 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:12.131439+0000 2024-04-25T21:40:12.130 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:12.130 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.130 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:12.130 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:12.130 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:12.130 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 1 Processor -- start 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 1 -- start start 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient: init 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f94f0072a50 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f94f00a15d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f94f00a1a20 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:12.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f0099740 con 0x7f94f00a15d0 2024-04-25T21:40:12.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f00734b0 con 0x7f94f00a1a20 2024-04-25T21:40:12.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f009bfe0 con 0x7f94f0072a50 2024-04-25T21:40:12.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:12.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94f4f6d640 10 monclient(hunting): authenticate will time out at 971.041430s 2024-04-25T21:40:12.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94edd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f94f00a15d0 0x7f94f013be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60710/0 (socket says 172.21.15.6:60710) 2024-04-25T21:40:12.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94edd9b640 1 -- 172.21.15.6:0/3104616866 learned_addr learned my addr 172.21.15.6:0/3104616866 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:12.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94edd9b640 1 -- 172.21.15.6:0/3104616866 >> v1:172.21.15.6:6789/0 conn(0x7f94f00a15d0 legacy=0x7f94f013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94ee59c640 1 -- 172.21.15.6:0/3104616866 >> v1:172.21.15.6:6790/0 conn(0x7f94f0072a50 legacy=0x7f94f00a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985190526 0 0) 0x7f94f0099740 con 0x7f94f00a15d0 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94ed59a640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94ed59a640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94ed59a640 10 monclient(hunting): my global_id is 4389 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.128+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94d8003530 con 0x7f94f00a15d0 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417567751 0 0) 0x7f94f009bfe0 con 0x7f94f0072a50 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient(hunting): my global_id is 4367 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f0099740 con 0x7f94f0072a50 2024-04-25T21:40:12.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94eed9d640 1 -- 172.21.15.6:0/3104616866 >> v1:172.21.15.203:6789/0 conn(0x7f94f00a1a20 legacy=0x7f94f013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1261761318 0 0) 0x7f94d8003530 con 0x7f94f00a15d0 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 >> v1:172.21.15.6:6790/0 conn(0x7f94f0072a50 legacy=0x7f94f00a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 >> v1:172.21.15.203:6789/0 conn(0x7f94f00a1a20 legacy=0x7f94f013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: _finish_hunting 0 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 monclient: found mon.noname-a 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f0099fc0 con 0x7f94f00a15d0 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: _finish_auth 0 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: _check_auth_tickets 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f94e4001990 con 0x7f94f00a15d0 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 5 monclient: authenticate success, global_id 4389 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:12.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: dump: 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94e4001c40 con 0x7f94f00a15d0 2024-04-25T21:40:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 4 set_mon_vals no callback set 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 1 -- 172.21.15.6:0/3104616866 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f94e4004a50 con 0x7f94f00a15d0 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ed59a640 10 monclient: dump: 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient: shutdown 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- 172.21.15.6:0/3104616866 >> v1:172.21.15.6:6789/0 conn(0x7f94f00a15d0 legacy=0x7f94f013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- 172.21.15.6:0/3104616866 shutdown_connections 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- 172.21.15.6:0/3104616866 >> 172.21.15.6:0/3104616866 conn(0x7f94f00a2180 msgr2=0x7f94f00a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- 172.21.15.6:0/3104616866 shutdown_connections 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- 172.21.15.6:0/3104616866 wait complete. 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient: build_initial_monmap 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient: monmap: 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 Processor -- start 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- start start 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient: init 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient(hunting): picked mon.noname-c con 0x7f94f0072a50 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient(hunting): picked mon.noname-a con 0x7f94f00a15d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient(hunting): picked mon.noname-b con 0x7f94f00a1a20 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f009e620 con 0x7f94f00a15d0 2024-04-25T21:40:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f0132390 con 0x7f94f00a1a20 2024-04-25T21:40:12.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94f009b940 con 0x7f94f0072a50 2024-04-25T21:40:12.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:12.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94f4f6d640 10 monclient(hunting): authenticate will time out at 971.046326s 2024-04-25T21:40:12.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94edd9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f94f00a15d0 0x7f94f009df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60714/0 (socket says 172.21.15.6:60714) 2024-04-25T21:40:12.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94ee59c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f94f0072a50 0x7f94f009d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33526/0 (socket says 172.21.15.6:33526) 2024-04-25T21:40:12.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.132+0000 7f94edd9b640 1 -- 172.21.15.6:0/4145799214 learned_addr learned my addr 172.21.15.6:0/4145799214 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:12.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94edd9b640 1 -- 172.21.15.6:0/4145799214 >> v1:172.21.15.6:6789/0 conn(0x7f94f00a15d0 legacy=0x7f94f009df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94ee59c640 1 -- 172.21.15.6:0/4145799214 >> v1:172.21.15.6:6790/0 conn(0x7f94f0072a50 legacy=0x7f94f009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046267105 0 0) 0x7f94f009e620 con 0x7f94f00a15d0 2024-04-25T21:40:12.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): my global_id is 4392 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94d0003530 con 0x7f94f00a15d0 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2069236923 0 0) 0x7f94f009b940 con 0x7f94f0072a50 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): my global_id is 4370 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f009e620 con 0x7f94f0072a50 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94eed9d640 1 -- 172.21.15.6:0/4145799214 >> v1:172.21.15.203:6789/0 conn(0x7f94f00a1a20 legacy=0x7f94f01d4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3445036522 0 0) 0x7f94d0003530 con 0x7f94f00a15d0 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94f009b940 con 0x7f94f00a15d0 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3074080181 0 0) 0x7f94f009e620 con 0x7f94f0072a50 2024-04-25T21:40:12.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94d0003530 con 0x7f94f0072a50 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f94e4004990 con 0x7f94f00a15d0 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): dump: 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f94e0001c20 con 0x7f94f0072a50 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): dump: 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1875545922 0 0) 0x7f94f0132390 con 0x7f94f00a1a20 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient(hunting): my global_id is 4333 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f009e620 con 0x7f94f00a1a20 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 437954968 0 0) 0x7f94f009b940 con 0x7f94f00a15d0 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 >> v1:172.21.15.6:6790/0 conn(0x7f94f0072a50 legacy=0x7f94f009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 >> v1:172.21.15.203:6789/0 conn(0x7f94f00a1a20 legacy=0x7f94f01d4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 monclient: found mon.a 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f009f3c0 con 0x7f94f00a15d0 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:12.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 5 monclient: authenticate success, global_id 4392 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 10 monclient: _renew_subs 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f94f0133f00 con 0x7f94f00a15d0 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 10 monclient: _renew_subs 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f94f0134670 con 0x7f94f00a15d0 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f94e4001c40 con 0x7f94f00a15d0 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f94e4005170 con 0x7f94f00a15d0 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94ecd99640 4 set_mon_vals no callback set 2024-04-25T21:40:12.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 10 monclient: dump: 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d4005090 con 0x7f94f00a15d0 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f94e4005350 con 0x7f94f00a15d0 2024-04-25T21:40:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f94e4044500 con 0x7f94f00a15d0 2024-04-25T21:40:12.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.136+0000 7f94ee59c640 10 monclient: get_auth_request con 0x7f94d0036c90 auth_method 0 2024-04-25T21:40:12.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.140+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f94e40158e0 con 0x7f94f00a15d0 2024-04-25T21:40:12.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.140+0000 7f94deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.140+0000 7f94deffd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.300+0000 7f94f4f6d640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-25T21:40:12.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.300+0000 7f94f4f6d640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-25T21:40:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.300+0000 7f94f4f6d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.300+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f94d4005b90 con 0x7f94f00a15d0 2024-04-25T21:40:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.300+0000 7f94deffd640 1 -- 172.21.15.6:0/4145799214 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) ==== 92+0+10 (unknown 1398289234 0 265718887) 0x7f94e40151e0 con 0x7f94f00a15d0 2024-04-25T21:40:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.300+0000 7f94deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-25T21:40:12.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.300+0000 7f94deffd640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:12.306 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 8 2024-04-25T21:40:12.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 >> v1:172.21.15.203:6812/3164820986 conn(0x7f94d0036c90 legacy=0x7f94d0039150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 10 monclient: shutdown 2024-04-25T21:40:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 >> v1:172.21.15.6:6789/0 conn(0x7f94f00a15d0 legacy=0x7f94f009df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 shutdown_connections 2024-04-25T21:40:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 >> 172.21.15.6:0/4145799214 conn(0x7f94f00a2180 msgr2=0x7f94f013fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 shutdown_connections 2024-04-25T21:40:12.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.304+0000 7f94f4f6d640 1 -- 172.21.15.6:0/4145799214 wait complete. 2024-04-25T21:40:12.322 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-04-25T21:40:12.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:12.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.436+0000 7f4f4e496640 10 monclient: get_monmap_and_config 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.436+0000 7f4f4e496640 10 monclient: build_initial_monmap 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.436+0000 7f4f4e496640 10 monclient: monmap: 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:12.442776+0000 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:12.442776+0000 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:12.441 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.436+0000 7f4f4e496640 1 Processor -- start 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.436+0000 7f4f4e496640 1 -- start start 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.436+0000 7f4f4e496640 10 monclient: init 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4809e860 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480a1270 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient(hunting): picked mon.noname-a con 0x7f4f480a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4809c130 con 0x7f4f480a1680 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f48099520 con 0x7f4f480a1270 2024-04-25T21:40:12.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f48099b10 con 0x7f4f4809e860 2024-04-25T21:40:12.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:12.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient(hunting): authenticate will time out at 971.352692s 2024-04-25T21:40:12.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f47fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4f4809e860 0x7f4f480a0b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33532/0 (socket says 172.21.15.6:33532) 2024-04-25T21:40:12.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f47fff640 1 -- 172.21.15.6:0/1990849934 learned_addr learned my addr 172.21.15.6:0/1990849934 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:12.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4ca33640 1 --1- 172.21.15.6:0/1990849934 >> v1:172.21.15.6:6789/0 conn(0x7f4f480a1680 0x7f4f4813d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60724/0 (socket says 172.21.15.6:60724) 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f47fff640 1 -- 172.21.15.6:0/1990849934 >> v1:172.21.15.6:6790/0 conn(0x7f4f4809e860 legacy=0x7f4f480a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4ca33640 1 -- 172.21.15.6:0/1990849934 >> v1:172.21.15.6:6789/0 conn(0x7f4f480a1680 legacy=0x7f4f4813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1139494569 0 0) 0x7f4f48099b10 con 0x7f4f4809e860 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): my global_id is 4373 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f28003530 con 0x7f4f4809e860 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777364142 0 0) 0x7f4f38001460 con 0x7f4f480a1680 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient(hunting): my global_id is 4395 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f477fe640 1 -- 172.21.15.6:0/1990849934 >> v1:172.21.15.203:6789/0 conn(0x7f4f480a1270 legacy=0x7f4f48139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f48099b10 con 0x7f4f480a1680 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3920636891 0 0) 0x7f4f28003530 con 0x7f4f4809e860 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 >> v1:172.21.15.203:6789/0 conn(0x7f4f480a1270 legacy=0x7f4f48139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 >> v1:172.21.15.6:6789/0 conn(0x7f4f480a1680 legacy=0x7f4f4813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 monclient: found mon.noname-b 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4809a0d0 con 0x7f4f4809e860 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:12.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4f40001c50 con 0x7f4f4809e860 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: dump: 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 5 monclient: authenticate success, global_id 4373 2024-04-25T21:40:12.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f40001fa0 con 0x7f4f4809e860 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 1 -- 172.21.15.6:0/1990849934 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4f40004dd0 con 0x7f4f4809e860 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f46ffd640 10 monclient: dump: 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.449 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.440+0000 7f4f4e496640 4 set_mon_vals no callback set 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient: shutdown 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- 172.21.15.6:0/1990849934 >> v1:172.21.15.6:6790/0 conn(0x7f4f4809e860 legacy=0x7f4f480a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- 172.21.15.6:0/1990849934 shutdown_connections 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- 172.21.15.6:0/1990849934 >> 172.21.15.6:0/1990849934 conn(0x7f4f480a1c50 msgr2=0x7f4f480a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- 172.21.15.6:0/1990849934 shutdown_connections 2024-04-25T21:40:12.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- 172.21.15.6:0/1990849934 wait complete. 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient: build_initial_monmap 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient: monmap: 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:12.454 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 Processor -- start 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- start start 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient: init 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4809e860 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient(hunting): picked mon.noname-a con 0x7f4f480a1270 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480a1680 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f48091d10 con 0x7f4f480a1270 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4809d470 con 0x7f4f4809e860 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f48092420 con 0x7f4f480a1680 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f4e496640 10 monclient(hunting): authenticate will time out at 971.358292s 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f477fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f480a1270 0x7f4f4809cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60730/0 (socket says 172.21.15.6:60730) 2024-04-25T21:40:12.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.444+0000 7f4f477fe640 1 -- 172.21.15.6:0/571182156 learned_addr learned my addr 172.21.15.6:0/571182156 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4ca33640 1 -- 172.21.15.6:0/571182156 >> v1:172.21.15.6:6790/0 conn(0x7f4f480a1680 legacy=0x7f4f481d3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f477fe640 1 -- 172.21.15.6:0/571182156 >> v1:172.21.15.6:6789/0 conn(0x7f4f480a1270 legacy=0x7f4f4809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988839401 0 0) 0x7f4f48091d10 con 0x7f4f480a1270 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f47fff640 1 -- 172.21.15.6:0/571182156 >> v1:172.21.15.203:6789/0 conn(0x7f4f4809e860 legacy=0x7f4f4809c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): my global_id is 4398 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c003530 con 0x7f4f480a1270 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510148697 0 0) 0x7f4f48092420 con 0x7f4f480a1680 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): my global_id is 4376 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f48091d10 con 0x7f4f480a1680 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894263189 0 0) 0x7f4f4809d470 con 0x7f4f4809e860 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): my global_id is 4339 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f48092420 con 0x7f4f4809e860 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1951713412 0 0) 0x7f4f2c003530 con 0x7f4f480a1270 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f4809d470 con 0x7f4f480a1270 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4f30001ba0 con 0x7f4f480a1270 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3899583114 0 0) 0x7f4f48091d10 con 0x7f4f480a1680 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f2c003530 con 0x7f4f480a1680 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4f380013f0 con 0x7f4f480a1680 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.459 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1167551931 0 0) 0x7f4f4809d470 con 0x7f4f480a1270 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 >> v1:172.21.15.6:6790/0 conn(0x7f4f480a1680 legacy=0x7f4f481d3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 >> v1:172.21.15.203:6789/0 conn(0x7f4f4809e860 legacy=0x7f4f4809c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 monclient: found mon.a 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f48072b40 con 0x7f4f480a1270 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 5 monclient: authenticate success, global_id 4398 2024-04-25T21:40:12.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 10 monclient: _renew_subs 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f480920e0 con 0x7f4f480a1270 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 10 monclient: _renew_subs 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f48090db0 con 0x7f4f480a1270 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f48062e50 con 0x7f4f480a1270 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f300038d0 con 0x7f4f480a1270 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4f30003c20 con 0x7f4f480a1270 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:12.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 10 monclient: dump: 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f4f30003dc0 con 0x7f4f480a1270 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f45ffb640 4 set_mon_vals no callback set 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f4f30044250 con 0x7f4f480a1270 2024-04-25T21:40:12.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.448+0000 7f4f47fff640 10 monclient: get_auth_request con 0x7f4f2c036c50 auth_method 0 2024-04-25T21:40:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.452+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f4f300155d0 con 0x7f4f480a1270 2024-04-25T21:40:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.452+0000 7f4f44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.452+0000 7f4f44ff9640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:12.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.608+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4f4806b2e0 con 0x7f4f2c036c50 2024-04-25T21:40:12.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f44ff9640 1 -- 172.21.15.6:0/571182156 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33782 (unknown 2965378022 0 1225781016) 0x7f4f4806b2e0 con 0x7f4f2c036c50 2024-04-25T21:40:12.615 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:12.615 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:12.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 >> v1:172.21.15.203:6812/3164820986 conn(0x7f4f2c036c50 legacy=0x7f4f2c039110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 10 monclient: shutdown 2024-04-25T21:40:12.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:12.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 >> v1:172.21.15.6:6789/0 conn(0x7f4f480a1270 legacy=0x7f4f4809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 shutdown_connections 2024-04-25T21:40:12.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 >> 172.21.15.6:0/571182156 conn(0x7f4f480a1c50 msgr2=0x7f4f48098780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:12.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 shutdown_connections 2024-04-25T21:40:12.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.612+0000 7f4f4e496640 1 -- 172.21.15.6:0/571182156 wait complete. 2024-04-25T21:40:12.631 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-25T21:40:11.518466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164140,"kb_used_data":1992,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066900,"statfs":{"total":579820584960,"available":579652505600,"internally_reserved":0,"allocated":2039808,"data_stored":1355822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.061086"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118941+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:08.118941+0000","last_peered":"2024-04-25T21:40:08.118941+0000","last_clean":"2024-04-25T21:40:08.118941+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:08.118941+0000","last_undegraded":"2024-04-25T21:40:08.118941+0000","last_fullsized":"2024-04-25T21:40:08.118941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122641+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:08.122641+0000","last_peered":"2024-04-25T21:40:08.122641+0000","last_clean":"2024-04-25T21:40:08.122641+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:08.122641+0000","last_undegraded":"2024-04-25T21:40:08.122641+0000","last_fullsized":"2024-04-25T21:40:08.122641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118903+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:08.118903+0000","last_peered":"2024-04-25T21:40:08.118903+0000","last_clean":"2024-04-25T21:40:08.118903+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:08.118903+0000","last_undegraded":"2024-04-25T21:40:08.118903+0000","last_fullsized":"2024-04-25T21:40:08.118903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122575+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:08.122575+0000","last_peered":"2024-04-25T21:40:08.122575+0000","last_clean":"2024-04-25T21:40:08.122575+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:08.122575+0000","last_undegraded":"2024-04-25T21:40:08.122575+0000","last_fullsized":"2024-04-25T21:40:08.122575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500532+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:08.500532+0000","last_peered":"2024-04-25T21:40:08.500532+0000","last_clean":"2024-04-25T21:40:08.500532+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:08.500532+0000","last_undegraded":"2024-04-25T21:40:08.500532+0000","last_fullsized":"2024-04-25T21:40:08.500532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.120646+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:08.120646+0000","last_peered":"2024-04-25T21:40:08.120646+0000","last_clean":"2024-04-25T21:40:08.120646+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:08.120646+0000","last_undegraded":"2024-04-25T21:40:08.120646+0000","last_fullsized":"2024-04-25T21:40:08.120646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118915+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:08.118915+0000","last_peered":"2024-04-25T21:40:08.118915+0000","last_clean":"2024-04-25T21:40:08.118915+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:08.118915+0000","last_undegraded":"2024-04-25T21:40:08.118915+0000","last_fullsized":"2024-04-25T21:40:08.118915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500564+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:08.500564+0000","last_peered":"2024-04-25T21:40:08.500564+0000","last_clean":"2024-04-25T21:40:08.500564+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:08.500564+0000","last_undegraded":"2024-04-25T21:40:08.500564+0000","last_fullsized":"2024-04-25T21:40:08.500564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118784+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:08.118784+0000","last_peered":"2024-04-25T21:40:08.118784+0000","last_clean":"2024-04-25T21:40:08.118784+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:08.118784+0000","last_undegraded":"2024-04-25T21:40:08.118784+0000","last_fullsized":"2024-04-25T21:40:08.118784+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":579263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":575219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:12.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.748+0000 7fec28730640 10 monclient: get_monmap_and_config 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.748+0000 7fec28730640 10 monclient: build_initial_monmap 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.748+0000 7fec28730640 10 monclient: monmap: 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:12.755074+0000 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:12.755074+0000 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:12.754 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:12.754 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:12.754 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 1 Processor -- start 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 1 -- start start 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient: init 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient(hunting): picked mon.noname-c con 0x7fec200a2c10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient(hunting): picked mon.noname-a con 0x7fec20139e50 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient(hunting): picked mon.noname-b con 0x7fec2013d960 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec200a0e40 con 0x7fec20139e50 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec200a1400 con 0x7fec200a2c10 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec200a3700 con 0x7fec2013d960 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec28730640 10 monclient(hunting): authenticate will time out at 971.665099s 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec25ccb640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fec20139e50 0x7fec2013c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60742/0 (socket says 172.21.15.6:60742) 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec26ccd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fec2013d960 0x7fec2013fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33538/0 (socket says 172.21.15.6:33538) 2024-04-25T21:40:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec25ccb640 1 -- 172.21.15.6:0/4131591186 learned_addr learned my addr 172.21.15.6:0/4131591186 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec26ccd640 1 -- 172.21.15.6:0/4131591186 >> v1:172.21.15.6:6790/0 conn(0x7fec2013d960 legacy=0x7fec2013fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec25ccb640 1 -- 172.21.15.6:0/4131591186 >> v1:172.21.15.6:6789/0 conn(0x7fec20139e50 legacy=0x7fec2013c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec264cc640 1 -- 172.21.15.6:0/4131591186 >> v1:172.21.15.203:6789/0 conn(0x7fec200a2c10 legacy=0x7fec200a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297390236 0 0) 0x7fec200a3700 con 0x7fec2013d960 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): my global_id is 4379 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec0c003530 con 0x7fec2013d960 2024-04-25T21:40:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2801312809 0 0) 0x7fec200a0e40 con 0x7fec20139e50 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 10 monclient(hunting): my global_id is 4401 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.752+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec200a3700 con 0x7fec20139e50 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 650849881 0 0) 0x7fec0c003530 con 0x7fec2013d960 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 >> v1:172.21.15.203:6789/0 conn(0x7fec200a2c10 legacy=0x7fec200a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 >> v1:172.21.15.6:6789/0 conn(0x7fec20139e50 legacy=0x7fec2013c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: _finish_hunting 0 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 monclient: found mon.noname-b 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2006e9f0 con 0x7fec2013d960 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: _finish_auth 0 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: _check_auth_tickets 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fec18001c50 con 0x7fec2013d960 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: dump: 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 5 monclient: authenticate success, global_id 4379 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec18001fa0 con 0x7fec2013d960 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 1 -- 172.21.15.6:0/4131591186 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fec18004d10 con 0x7fec2013d960 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec254ca640 10 monclient: dump: 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 4 set_mon_vals no callback set 2024-04-25T21:40:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient: shutdown 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- 172.21.15.6:0/4131591186 >> v1:172.21.15.6:6790/0 conn(0x7fec2013d960 legacy=0x7fec2013fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- 172.21.15.6:0/4131591186 shutdown_connections 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- 172.21.15.6:0/4131591186 >> 172.21.15.6:0/4131591186 conn(0x7fec200a5e10 msgr2=0x7fec200a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- 172.21.15.6:0/4131591186 shutdown_connections 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- 172.21.15.6:0/4131591186 wait complete. 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient: build_initial_monmap 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient: monmap: 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 Processor -- start 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- start start 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient: init 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient(hunting): picked mon.noname-b con 0x7fec200a2c10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient(hunting): picked mon.noname-a con 0x7fec20139e50 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient(hunting): picked mon.noname-c con 0x7fec2013d960 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec2013b7a0 con 0x7fec20139e50 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec201305b0 con 0x7fec200a2c10 2024-04-25T21:40:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec200a5930 con 0x7fec2013d960 2024-04-25T21:40:12.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:12.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec28730640 10 monclient(hunting): authenticate will time out at 971.670344s 2024-04-25T21:40:12.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec25ccb640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fec20139e50 0x7fec2012fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60750/0 (socket says 172.21.15.6:60750) 2024-04-25T21:40:12.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.756+0000 7fec25ccb640 1 -- 172.21.15.6:0/2189466755 learned_addr learned my addr 172.21.15.6:0/2189466755 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec25ccb640 1 -- 172.21.15.6:0/2189466755 >> v1:172.21.15.6:6789/0 conn(0x7fec20139e50 legacy=0x7fec2012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec26ccd640 1 -- 172.21.15.6:0/2189466755 >> v1:172.21.15.6:6790/0 conn(0x7fec2013d960 legacy=0x7fec201d3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec264cc640 1 -- 172.21.15.6:0/2189466755 >> v1:172.21.15.203:6789/0 conn(0x7fec200a2c10 legacy=0x7fec2012f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149629491 0 0) 0x7fec2013b7a0 con 0x7fec20139e50 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): my global_id is 4404 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec04003530 con 0x7fec20139e50 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 406056437 0 0) 0x7fec200a5930 con 0x7fec2013d960 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): my global_id is 4382 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec2013b7a0 con 0x7fec2013d960 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1207833773 0 0) 0x7fec201305b0 con 0x7fec200a2c10 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): my global_id is 4345 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec200a5930 con 0x7fec200a2c10 2024-04-25T21:40:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4023572228 0 0) 0x7fec04003530 con 0x7fec20139e50 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec201305b0 con 0x7fec20139e50 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fec100015e0 con 0x7fec20139e50 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.772 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2179594682 0 0) 0x7fec2013b7a0 con 0x7fec2013d960 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec04003530 con 0x7fec2013d960 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fec18004d10 con 0x7fec2013d960 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.773 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1197407474 0 0) 0x7fec04003530 con 0x7fec2013d960 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 >> v1:172.21.15.203:6789/0 conn(0x7fec200a2c10 legacy=0x7fec2012f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 >> v1:172.21.15.6:6789/0 conn(0x7fec20139e50 legacy=0x7fec2012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 monclient: found mon.c 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec20131100 con 0x7fec2013d960 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:12.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 5 monclient: authenticate success, global_id 4382 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 10 monclient: _renew_subs 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec2013b170 con 0x7fec2013d960 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 10 monclient: _renew_subs 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fec2013ae20 con 0x7fec2013d960 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec18001fa0 con 0x7fec2013d960 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fec18005520 con 0x7fec2013d960 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec24cc9640 4 set_mon_vals no callback set 2024-04-25T21:40:12.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 10 monclient: dump: 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec08005090 con 0x7fec2013d960 2024-04-25T21:40:12.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fec180102c0 con 0x7fec2013d960 2024-04-25T21:40:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7fec180446c0 con 0x7fec2013d960 2024-04-25T21:40:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.760+0000 7fec264cc640 10 monclient: get_auth_request con 0x7fec04036b70 auth_method 0 2024-04-25T21:40:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.764+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fec18015aa0 con 0x7fec2013d960 2024-04-25T21:40:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.764+0000 7fec16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:12.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.764+0000 7fec16ffd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:12.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.916+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec08000ba0 con 0x7fec04036b70 2024-04-25T21:40:12.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.920+0000 7fec16ffd640 1 -- 172.21.15.6:0/2189466755 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33782 (unknown 2965378022 0 1225781016) 0x7fec08000ba0 con 0x7fec04036b70 2024-04-25T21:40:12.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:12.925 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:12.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 >> v1:172.21.15.203:6812/3164820986 conn(0x7fec04036b70 legacy=0x7fec04039030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 10 monclient: shutdown 2024-04-25T21:40:12.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:12.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 >> v1:172.21.15.6:6790/0 conn(0x7fec2013d960 legacy=0x7fec201d3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:12.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 shutdown_connections 2024-04-25T21:40:12.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 >> 172.21.15.6:0/2189466755 conn(0x7fec200a5e10 msgr2=0x7fec20133f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:12.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 shutdown_connections 2024-04-25T21:40:12.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:12.924+0000 7fec28730640 1 -- 172.21.15.6:0/2189466755 wait complete. 2024-04-25T21:40:12.941 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-25T21:40:11.518466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164140,"kb_used_data":1992,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066900,"statfs":{"total":579820584960,"available":579652505600,"internally_reserved":0,"allocated":2039808,"data_stored":1355822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.061086"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118941+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:08.118941+0000","last_peered":"2024-04-25T21:40:08.118941+0000","last_clean":"2024-04-25T21:40:08.118941+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:08.118941+0000","last_undegraded":"2024-04-25T21:40:08.118941+0000","last_fullsized":"2024-04-25T21:40:08.118941+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122641+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:08.122641+0000","last_peered":"2024-04-25T21:40:08.122641+0000","last_clean":"2024-04-25T21:40:08.122641+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:08.122641+0000","last_undegraded":"2024-04-25T21:40:08.122641+0000","last_fullsized":"2024-04-25T21:40:08.122641+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118903+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:08.118903+0000","last_peered":"2024-04-25T21:40:08.118903+0000","last_clean":"2024-04-25T21:40:08.118903+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:08.118903+0000","last_undegraded":"2024-04-25T21:40:08.118903+0000","last_fullsized":"2024-04-25T21:40:08.118903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.122575+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:08.122575+0000","last_peered":"2024-04-25T21:40:08.122575+0000","last_clean":"2024-04-25T21:40:08.122575+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:08.122575+0000","last_undegraded":"2024-04-25T21:40:08.122575+0000","last_fullsized":"2024-04-25T21:40:08.122575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500532+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:08.500532+0000","last_peered":"2024-04-25T21:40:08.500532+0000","last_clean":"2024-04-25T21:40:08.500532+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:08.500532+0000","last_undegraded":"2024-04-25T21:40:08.500532+0000","last_fullsized":"2024-04-25T21:40:08.500532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.120646+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:08.120646+0000","last_peered":"2024-04-25T21:40:08.120646+0000","last_clean":"2024-04-25T21:40:08.120646+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:08.120646+0000","last_undegraded":"2024-04-25T21:40:08.120646+0000","last_fullsized":"2024-04-25T21:40:08.120646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118915+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:08.118915+0000","last_peered":"2024-04-25T21:40:08.118915+0000","last_clean":"2024-04-25T21:40:08.118915+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:08.118915+0000","last_undegraded":"2024-04-25T21:40:08.118915+0000","last_fullsized":"2024-04-25T21:40:08.118915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.500564+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:08.500564+0000","last_peered":"2024-04-25T21:40:08.500564+0000","last_clean":"2024-04-25T21:40:08.500564+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:08.500564+0000","last_undegraded":"2024-04-25T21:40:08.500564+0000","last_fullsized":"2024-04-25T21:40:08.500564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":168,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-04-25T21:40:08.118784+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:08.118784+0000","last_peered":"2024-04-25T21:40:08.118784+0000","last_clean":"2024-04-25T21:40:08.118784+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:08.118784+0000","last_undegraded":"2024-04-25T21:40:08.118784+0000","last_fullsized":"2024-04-25T21:40:08.118784+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":163840,"data_stored":50335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":579263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":575219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:12.942 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-04-25T21:40:12.942 DEBUG:tasks.mon_thrash:Yielding 2024-04-25T21:40:12.942 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T21:40:12.952 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T21:40:12.952 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:40:12.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20' 2024-04-25T21:40:12.952 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-04-25T21:40:12.952 INFO:tasks.mon_thrash.mon_thrasher:seed: 1714081212, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-04-25T21:40:12.953 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-25T21:40:12.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-25T21:40:13.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient: get_monmap_and_config 2024-04-25T21:40:13.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient: monmap: 2024-04-25T21:40:13.067 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.067 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:13.069517+0000 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:13.069517+0000 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:13.068 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 1 Processor -- start 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 1 -- start start 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient: init 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient(hunting): picked mon.noname-c con 0x7f3b4809e860 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient(hunting): picked mon.noname-a con 0x7f3b480a1270 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient(hunting): picked mon.noname-b con 0x7f3b480a1680 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b4809c130 con 0x7f3b480a1270 2024-04-25T21:40:13.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b48099520 con 0x7f3b4809e860 2024-04-25T21:40:13.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b48099b10 con 0x7f3b480a1680 2024-04-25T21:40:13.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b4ef95640 10 monclient(hunting): authenticate will time out at 971.978933s 2024-04-25T21:40:13.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b47fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3b480a1270 0x7f3b48139b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60760/0 (socket says 172.21.15.6:60760) 2024-04-25T21:40:13.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.064+0000 7f3b47fff640 1 -- 172.21.15.6:0/644255886 learned_addr learned my addr 172.21.15.6:0/644255886 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b47fff640 1 -- 172.21.15.6:0/644255886 >> v1:172.21.15.6:6789/0 conn(0x7f3b480a1270 legacy=0x7f3b48139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4d532640 1 -- 172.21.15.6:0/644255886 >> v1:172.21.15.6:6790/0 conn(0x7f3b480a1680 legacy=0x7f3b4813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4cd31640 1 -- 172.21.15.6:0/644255886 >> v1:172.21.15.203:6789/0 conn(0x7f3b4809e860 legacy=0x7f3b480a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101494683 0 0) 0x7f3b3c001460 con 0x7f3b480a1270 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): my global_id is 4413 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b30003530 con 0x7f3b480a1270 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2914146649 0 0) 0x7f3b48099b10 con 0x7f3b480a1680 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): my global_id is 4397 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b3c001460 con 0x7f3b480a1680 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2937508938 0 0) 0x7f3b48099520 con 0x7f3b4809e860 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient(hunting): my global_id is 4354 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b48099b10 con 0x7f3b4809e860 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4144883818 0 0) 0x7f3b30003530 con 0x7f3b480a1270 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 >> v1:172.21.15.6:6790/0 conn(0x7f3b480a1680 legacy=0x7f3b4813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 >> v1:172.21.15.203:6789/0 conn(0x7f3b4809e860 legacy=0x7f3b480a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 monclient: found mon.noname-a 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4809a0d0 con 0x7f3b480a1270 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3b3c001aa0 con 0x7f3b480a1270 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 5 monclient: authenticate success, global_id 4413 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: dump: 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.075 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b3c0048b0 con 0x7f3b480a1270 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 1 -- 172.21.15.6:0/644255886 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3b3c004c00 con 0x7f3b480a1270 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b477fe640 10 monclient: dump: 2024-04-25T21:40:13.076 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 4 set_mon_vals no callback set 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 10 monclient: shutdown 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 1 -- 172.21.15.6:0/644255886 >> v1:172.21.15.6:6789/0 conn(0x7f3b480a1270 legacy=0x7f3b48139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 1 -- 172.21.15.6:0/644255886 shutdown_connections 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 1 -- 172.21.15.6:0/644255886 >> 172.21.15.6:0/644255886 conn(0x7f3b480a1c50 msgr2=0x7f3b480a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 1 -- 172.21.15.6:0/644255886 shutdown_connections 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 1 -- 172.21.15.6:0/644255886 wait complete. 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 10 monclient: monmap: 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:13.078 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 1 Processor -- start 2024-04-25T21:40:13.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 1 -- start start 2024-04-25T21:40:13.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.068+0000 7f3b4ef95640 10 monclient: init 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient(hunting): picked mon.noname-c con 0x7f3b4809e860 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient(hunting): picked mon.noname-b con 0x7f3b480a1270 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient(hunting): picked mon.noname-a con 0x7f3b480a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b48092170 con 0x7f3b480a1680 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b48097af0 con 0x7f3b480a1270 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b4808c270 con 0x7f3b4809e860 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient(hunting): authenticate will time out at 971.984310s 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4cd31640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f3b4809e860 0x7f3b4813d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33546/0 (socket says 172.21.15.6:33546) 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4d532640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3b480a1680 0x7f3b481d2460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60772/0 (socket says 172.21.15.6:60772) 2024-04-25T21:40:13.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4cd31640 1 -- 172.21.15.6:0/3813596579 learned_addr learned my addr 172.21.15.6:0/3813596579 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4d532640 1 -- 172.21.15.6:0/3813596579 >> v1:172.21.15.6:6789/0 conn(0x7f3b480a1680 legacy=0x7f3b481d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4cd31640 1 -- 172.21.15.6:0/3813596579 >> v1:172.21.15.6:6790/0 conn(0x7f3b4809e860 legacy=0x7f3b4813d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538012262 0 0) 0x7f3b4808c270 con 0x7f3b4809e860 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): my global_id is 4400 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b34003530 con 0x7f3b4809e860 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b47fff640 1 -- 172.21.15.6:0/3813596579 >> v1:172.21.15.203:6789/0 conn(0x7f3b480a1270 legacy=0x7f3b481ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 317641108 0 0) 0x7f3b34003530 con 0x7f3b4809e860 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3b4808c270 con 0x7f3b4809e860 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3b38001c50 con 0x7f3b4809e860 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:13.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): dump: 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1540795467 0 0) 0x7f3b4808c270 con 0x7f3b4809e860 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 >> v1:172.21.15.203:6789/0 conn(0x7f3b480a1270 legacy=0x7f3b481ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 >> v1:172.21.15.6:6789/0 conn(0x7f3b480a1680 legacy=0x7f3b481d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 monclient: found mon.c 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4808d160 con 0x7f3b4809e860 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 5 monclient: authenticate success, global_id 4400 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient: _renew_subs 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4808ce30 con 0x7f3b4809e860 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient: _renew_subs 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3b48096ee0 con 0x7f3b4809e860 2024-04-25T21:40:13.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3b38003ad0 con 0x7f3b4809e860 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3b38003e20 con 0x7f3b4809e860 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 10 monclient: dump: 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.087 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b46ffd640 4 set_mon_vals no callback set 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.072+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f3b38004000 con 0x7f3b4809e860 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.076+0000 7f3b4ef95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.076+0000 7f3b4ef95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.076+0000 7f3b4ef95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.076+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b28005090 con 0x7f3b4809e860 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.076+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f3b380444b0 con 0x7f3b4809e860 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.076+0000 7f3b47fff640 10 monclient: get_auth_request con 0x7f3b34036a70 auth_method 0 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.080+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f3b38015830 con 0x7f3b4809e860 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.080+0000 7f3b457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.080+0000 7f3b457fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:13.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.296+0000 7f3b4ef95640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-25T21:40:13.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.296+0000 7f3b4ef95640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:13.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.296+0000 7f3b4ef95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.296+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3b28005380 con 0x7f3b4809e860 2024-04-25T21:40:13.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.296+0000 7f3b457fa640 1 -- 172.21.15.6:0/3813596579 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1401 (unknown 254746657 0 945416312) 0x7f3b38015130 con 0x7f3b4809e860 2024-04-25T21:40:13.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.296+0000 7f3b457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:13.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.296+0000 7f3b457fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:13.302 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","modified":"2024-04-25T21:39:14.743126Z","created":"2024-04-25T21:39:14.743126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-25T21:40:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 >> v1:172.21.15.203:6812/3164820986 conn(0x7f3b34036a70 legacy=0x7f3b34038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 10 monclient: shutdown 2024-04-25T21:40:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 >> v1:172.21.15.6:6790/0 conn(0x7f3b4809e860 legacy=0x7f3b4813d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 shutdown_connections 2024-04-25T21:40:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 >> 172.21.15.6:0/3813596579 conn(0x7f3b480a1c50 msgr2=0x7f3b48098880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 shutdown_connections 2024-04-25T21:40:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.300+0000 7f3b4ef95640 1 -- 172.21.15.6:0/3813596579 wait complete. 2024-04-25T21:40:13.317 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-25T21:40:13.318 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-25T21:40:13.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-25T21:40:13.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-25T21:40:13.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.432+0000 7f948baf5640 10 monclient: get_monmap_and_config 2024-04-25T21:40:13.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.432+0000 7f948baf5640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.432+0000 7f948baf5640 10 monclient: monmap: 2024-04-25T21:40:13.437 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.437 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.437 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:13.439480+0000 2024-04-25T21:40:13.438 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:13.439480+0000 2024-04-25T21:40:13.438 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.438 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.438 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.438 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:13.438 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:13.438 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 1 Processor -- start 2024-04-25T21:40:13.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 1 -- start start 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient: init 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient(hunting): picked mon.noname-b con 0x7f94840a2c20 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient(hunting): picked mon.noname-c con 0x7f94840a40d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient(hunting): picked mon.noname-a con 0x7f948413d580 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94840a0e50 con 0x7f948413d580 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94840a1130 con 0x7f94840a40d0 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94840a3710 con 0x7f94840a2c20 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948baf5640 10 monclient(hunting): authenticate will time out at 972.349672s 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948a092640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f948413d580 0x7f948413f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60782/0 (socket says 172.21.15.6:60782) 2024-04-25T21:40:13.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.436+0000 7f948a092640 1 -- 172.21.15.6:0/545097850 learned_addr learned my addr 172.21.15.6:0/545097850 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948a092640 1 -- 172.21.15.6:0/545097850 >> v1:172.21.15.6:6789/0 conn(0x7f948413d580 legacy=0x7f948413f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f9489891640 1 -- 172.21.15.6:0/545097850 >> v1:172.21.15.6:6790/0 conn(0x7f94840a2c20 legacy=0x7f94840a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847845291 0 0) 0x7f94840a0e50 con 0x7f948413d580 2024-04-25T21:40:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): my global_id is 4416 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9474003530 con 0x7f948413d580 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f9489090640 1 -- 172.21.15.6:0/545097850 >> v1:172.21.15.203:6789/0 conn(0x7f94840a40d0 legacy=0x7f948413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3132826588 0 0) 0x7f94840a3710 con 0x7f94840a2c20 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient(hunting): my global_id is 4403 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94840a0e50 con 0x7f94840a2c20 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 617122501 0 0) 0x7f9474003530 con 0x7f948413d580 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 >> v1:172.21.15.6:6790/0 conn(0x7f94840a2c20 legacy=0x7f94840a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 >> v1:172.21.15.203:6789/0 conn(0x7f94840a40d0 legacy=0x7f948413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 monclient: found mon.noname-a 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948406e9f0 con 0x7f948413d580 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f947c0018d0 con 0x7f948413d580 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 5 monclient: authenticate success, global_id 4416 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: dump: 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.445 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f947c001c20 con 0x7f948413d580 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 1 -- 172.21.15.6:0/545097850 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f947c004a50 con 0x7f948413d580 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948888f640 10 monclient: dump: 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 4 set_mon_vals no callback set 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient: shutdown 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- 172.21.15.6:0/545097850 >> v1:172.21.15.6:6789/0 conn(0x7f948413d580 legacy=0x7f948413f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- 172.21.15.6:0/545097850 shutdown_connections 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- 172.21.15.6:0/545097850 >> 172.21.15.6:0/545097850 conn(0x7f94840a5e10 msgr2=0x7f94840a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- 172.21.15.6:0/545097850 shutdown_connections 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- 172.21.15.6:0/545097850 wait complete. 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient: monmap: 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:13.448 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 Processor -- start 2024-04-25T21:40:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- start start 2024-04-25T21:40:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient: init 2024-04-25T21:40:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient(hunting): picked mon.noname-b con 0x7f94840a2c20 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient(hunting): picked mon.noname-a con 0x7f94840a40d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient(hunting): picked mon.noname-c con 0x7f948413d580 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9484140530 con 0x7f94840a40d0 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9484122160 con 0x7f94840a2c20 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94841cc830 con 0x7f948413d580 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f948baf5640 10 monclient(hunting): authenticate will time out at 972.354783s 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f9489090640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f94840a40d0 0x7f94841d0cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60784/0 (socket says 172.21.15.6:60784) 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.440+0000 7f9489090640 1 -- 172.21.15.6:0/2813748895 learned_addr learned my addr 172.21.15.6:0/2813748895 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948a092640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6790/0 conn(0x7f948413d580 legacy=0x7f94841d43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f9489090640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f94840a40d0 legacy=0x7f94841d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948a092640 1 --1- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6790/0 conn(0x7f948413d580 0x7f94841d43e0 :-1 s=OPENED pgs=138 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f94841cc830 auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341148243 0 0) 0x7f94840a1130 con 0x7f94840a40d0 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): my global_id is 4419 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9468003530 con 0x7f94840a40d0 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f9489891640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.203:6789/0 conn(0x7f94840a2c20 legacy=0x7f94841cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1180176926 0 0) 0x7f9468003530 con 0x7f94840a40d0 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f94840a1130 con 0x7f94840a40d0 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9478001f20 con 0x7f94840a40d0 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): dump: 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1435156458 0 0) 0x7f9484122160 con 0x7f94840a2c20 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient(hunting): my global_id is 4363 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9468003530 con 0x7f94840a2c20 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1147970987 0 0) 0x7f94840a1130 con 0x7f94840a40d0 2024-04-25T21:40:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6790/0 conn(0x7f948413d580 legacy=0x7f94841d43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.203:6789/0 conn(0x7f94840a2c20 legacy=0x7f94841cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 monclient: found mon.a 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9484124450 con 0x7f94840a40d0 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 5 monclient: authenticate success, global_id 4419 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 979102826 0 0) 0x7f94841cc830 con 0x7f948413d580 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 10 monclient: _renew_subs 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9484123bb0 con 0x7f94840a40d0 2024-04-25T21:40:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 10 monclient: _renew_subs 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9484121520 con 0x7f94840a40d0 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9478003cd0 con 0x7f94840a40d0 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9478004020 con 0x7f94840a40d0 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f9483fff640 4 set_mon_vals no callback set 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 10 monclient: dump: 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f946c0065d0 con 0x7f946c002870 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f94780041c0 con 0x7f94840a40d0 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3506+0+0 (unknown 2355523539 0 0) 0x7f94780447d0 con 0x7f94840a40d0 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.444+0000 7f948a092640 10 monclient: get_auth_request con 0x7f9468036b00 auth_method 0 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f9489891640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f946c002870 legacy=0x7f946c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277481761 0 0) 0x7f946c0065d0 con 0x7f946c002870 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient(hunting): my global_id is 4422 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94841cc830 con 0x7f946c002870 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2501068611 0 0) 0x7f94841cc830 con 0x7f946c002870 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f946c0065d0 con 0x7f946c002870 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9470001740 con 0x7f946c002870 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient: dump: 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2407657512 0 0) 0x7f946c0065d0 con 0x7f946c002870 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f946c00a160 con 0x7f946c002870 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9470006860 con 0x7f946c002870 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f946c002870 legacy=0x7f946c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.448+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f946c002870 legacy=0x7f946c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f946c009e50 con 0x7f946c009110 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f9489891640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f946c009110 legacy=0x7f946c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1861416654 0 0) 0x7f946c009e50 con 0x7f946c009110 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient(hunting): my global_id is 4425 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f946c0065d0 con 0x7f946c009110 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 461434887 0 0) 0x7f946c0065d0 con 0x7f946c009110 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f946c009e50 con 0x7f946c009110 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f9470003ab0 con 0x7f946c009110 2024-04-25T21:40:13.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient: dump: 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1076064075 0 0) 0x7f946c009e50 con 0x7f946c009110 2024-04-25T21:40:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f946c00c3e0 con 0x7f946c009110 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4143 (unknown 0 0 3594404772) 0x7f9470006a10 con 0x7f946c009110 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f946c009110 legacy=0x7f946c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f94827fc640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f946c009110 legacy=0x7f946c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 12, 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 33627, 2024-04-25T21:40:13.465 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 27, 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.466 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.467 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:13.468 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.469 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.470 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.471 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.472 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:13.473 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.474 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.475 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:13.476 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.203:6812/3164820986 conn(0x7f9468036b00 legacy=0x7f9468038fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 10 monclient: shutdown 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 >> v1:172.21.15.6:6789/0 conn(0x7f94840a40d0 legacy=0x7f94841d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948a092640 1 -- 172.21.15.6:0/2813748895 reap_dead start 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948a092640 1 -- 172.21.15.6:0/2813748895 reap_dead start 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 shutdown_connections 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.456+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 >> 172.21.15.6:0/2813748895 conn(0x7f94840a5e10 msgr2=0x7f948406d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.460+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 shutdown_connections 2024-04-25T21:40:13.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.460+0000 7f948baf5640 1 -- 172.21.15.6:0/2813748895 wait complete. 2024-04-25T21:40:13.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-25T21:40:13.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-25T21:40:13.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-25T21:40:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient: get_monmap_and_config 2024-04-25T21:40:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient: monmap: 2024-04-25T21:40:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.590 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:13.592320+0000 2024-04-25T21:40:13.591 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:13.592320+0000 2024-04-25T21:40:13.591 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.591 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.591 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.591 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:13.591 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:13.591 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 1 Processor -- start 2024-04-25T21:40:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 1 -- start start 2024-04-25T21:40:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient: init 2024-04-25T21:40:13.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient(hunting): picked mon.noname-c con 0x7f2d8c0a0b60 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient(hunting): picked mon.noname-b con 0x7f2d8c09ed30 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient(hunting): picked mon.noname-a con 0x7f2d8c09f180 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8c09aa70 con 0x7f2d8c09f180 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8c072c40 con 0x7f2d8c0a0b60 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8c09bbf0 con 0x7f2d8c09ed30 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d9358c640 10 monclient(hunting): authenticate will time out at 972.502202s 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d91b29640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d8c09f180 0x7f2d8c13c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60818/0 (socket says 172.21.15.6:60818) 2024-04-25T21:40:13.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.588+0000 7f2d91b29640 1 -- 172.21.15.6:0/3339982027 learned_addr learned my addr 172.21.15.6:0/3339982027 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d91b29640 1 -- 172.21.15.6:0/3339982027 >> v1:172.21.15.6:6789/0 conn(0x7f2d8c09f180 legacy=0x7f2d8c13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d90b27640 1 -- 172.21.15.6:0/3339982027 >> v1:172.21.15.6:6790/0 conn(0x7f2d8c09ed30 legacy=0x7f2d8c138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2278595080 0 0) 0x7f2d8c09aa70 con 0x7f2d8c09f180 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient(hunting): my global_id is 4428 2024-04-25T21:40:13.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d78003530 con 0x7f2d8c09f180 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d91328640 1 -- 172.21.15.6:0/3339982027 >> v1:172.21.15.203:6789/0 conn(0x7f2d8c0a0b60 legacy=0x7f2d8c09a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1295094071 0 0) 0x7f2d78003530 con 0x7f2d8c09f180 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 >> v1:172.21.15.6:6790/0 conn(0x7f2d8c09ed30 legacy=0x7f2d8c138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 >> v1:172.21.15.203:6789/0 conn(0x7f2d8c0a0b60 legacy=0x7f2d8c09a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 monclient: found mon.noname-a 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8c09bed0 con 0x7f2d8c09f180 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2d84001c50 con 0x7f2d8c09f180 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 5 monclient: authenticate success, global_id 4428 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: dump: 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.598 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d84001fa0 con 0x7f2d8c09f180 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 1 -- 172.21.15.6:0/3339982027 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2d84004dd0 con 0x7f2d8c09f180 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d83fff640 10 monclient: dump: 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 4 set_mon_vals no callback set 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient: shutdown 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- 172.21.15.6:0/3339982027 >> v1:172.21.15.6:6789/0 conn(0x7f2d8c09f180 legacy=0x7f2d8c13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- 172.21.15.6:0/3339982027 shutdown_connections 2024-04-25T21:40:13.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- 172.21.15.6:0/3339982027 >> 172.21.15.6:0/3339982027 conn(0x7f2d8c0a1c50 msgr2=0x7f2d8c0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- 172.21.15.6:0/3339982027 shutdown_connections 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- 172.21.15.6:0/3339982027 wait complete. 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient: monmap: 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:13.601 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.602 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:13.602 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:13.602 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 Processor -- start 2024-04-25T21:40:13.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- start start 2024-04-25T21:40:13.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient: init 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient(hunting): picked mon.noname-c con 0x7f2d8c09ed30 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient(hunting): picked mon.noname-a con 0x7f2d8c09f180 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient(hunting): picked mon.noname-b con 0x7f2d8c0a0b60 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8c0909b0 con 0x7f2d8c09f180 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8c09b730 con 0x7f2d8c0a0b60 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d8c0964e0 con 0x7f2d8c09ed30 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d9358c640 10 monclient(hunting): authenticate will time out at 972.507379s 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d91328640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f2d8c09ed30 0x7f2d8c1cbbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33574/0 (socket says 172.21.15.6:33574) 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.592+0000 7f2d91328640 1 -- 172.21.15.6:0/3322737059 learned_addr learned my addr 172.21.15.6:0/3322737059 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.596+0000 7f2d91328640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d8c09ed30 legacy=0x7f2d8c1cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.596+0000 7f2d90b27640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6789/0 conn(0x7f2d8c09f180 legacy=0x7f2d8c1d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d91b29640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.203:6789/0 conn(0x7f2d8c0a0b60 legacy=0x7f2d8c1d4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322521459 0 0) 0x7f2d8c0909b0 con 0x7f2d8c09f180 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): my global_id is 4431 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d70003530 con 0x7f2d8c09f180 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d90b27640 1 --1- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6789/0 conn(0x7f2d8c09f180 0x7f2d8c1d0e60 :-1 s=OPENED pgs=146 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f2d70003530 auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3994853325 0 0) 0x7f2d70003530 con 0x7f2d8c09f180 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d8c0909b0 con 0x7f2d8c09f180 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2d7c001b00 con 0x7f2d8c09f180 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): dump: 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.607 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135361913 0 0) 0x7f2d8c09b730 con 0x7f2d8c0a0b60 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): my global_id is 4366 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d70003530 con 0x7f2d8c0a0b60 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311975298 0 0) 0x7f2d8c0964e0 con 0x7f2d8c09ed30 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): my global_id is 4409 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d8c09b730 con 0x7f2d8c09ed30 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3820107037 0 0) 0x7f2d8c09b730 con 0x7f2d8c09ed30 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d8c0964e0 con 0x7f2d8c09ed30 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2d88001b80 con 0x7f2d8c09ed30 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient(hunting): dump: 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.609 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1348907014 0 0) 0x7f2d8c0964e0 con 0x7f2d8c09ed30 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.203:6789/0 conn(0x7f2d8c0a0b60 legacy=0x7f2d8c1d4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6789/0 conn(0x7f2d8c09f180 legacy=0x7f2d8c1d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 monclient: found mon.c 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8c08e9a0 con 0x7f2d8c09ed30 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d9358c640 5 monclient: authenticate success, global_id 4409 2024-04-25T21:40:13.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d9358c640 10 monclient: _renew_subs 2024-04-25T21:40:13.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d9358c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8c092b30 con 0x7f2d8c09ed30 2024-04-25T21:40:13.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1512887891 0 0) 0x7f2d8c0909b0 con 0x7f2d8c09f180 2024-04-25T21:40:13.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:13.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2d880038d0 con 0x7f2d8c09ed30 2024-04-25T21:40:13.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2d88003c20 con 0x7f2d8c09ed30 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.600+0000 7f2d81ffb640 10 monclient: dump: 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d9358c640 10 monclient: _renew_subs 2024-04-25T21:40:13.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d9358c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d8c08dd10 con 0x7f2d8c09ed30 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d82ffd640 4 set_mon_vals no callback set 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f2d88003e40 con 0x7f2d8c09ed30 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3898+0+0 (unknown 2549805797 0 0) 0x7f2d88043350 con 0x7f2d8c09ed30 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d9358c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2d6c0065d0 con 0x7f2d6c002870 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d90b27640 10 monclient: get_auth_request con 0x7f2d70036f00 auth_method 0 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.604+0000 7f2d91b29640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d6c002870 legacy=0x7f2d6c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993038686 0 0) 0x7f2d6c0065d0 con 0x7f2d6c002870 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient(hunting): my global_id is 4412 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d8c0909b0 con 0x7f2d6c002870 2024-04-25T21:40:13.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4047471588 0 0) 0x7f2d8c0909b0 con 0x7f2d6c002870 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d6c0065d0 con 0x7f2d6c002870 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2d84004d10 con 0x7f2d6c002870 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient: dump: 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.616 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 376269820 0 0) 0x7f2d6c0065d0 con 0x7f2d6c002870 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d6c00a160 con 0x7f2d6c002870 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2d84004bd0 con 0x7f2d6c002870 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d6c002870 legacy=0x7f2d6c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.608+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d6c002870 legacy=0x7f2d6c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.612+0000 7f2d9358c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:13.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.612+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2d6c009e50 con 0x7f2d6c009110 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d91b29640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d6c009110 legacy=0x7f2d6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432609039 0 0) 0x7f2d6c009e50 con 0x7f2d6c009110 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient(hunting): my global_id is 4415 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d6c0065d0 con 0x7f2d6c009110 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2429513581 0 0) 0x7f2d6c0065d0 con 0x7f2d6c009110 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d6c009e50 con 0x7f2d6c009110 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2d84001ca0 con 0x7f2d6c009110 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient: dump: 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 641340374 0 0) 0x7f2d6c009e50 con 0x7f2d6c009110 2024-04-25T21:40:13.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d6c00c3e0 con 0x7f2d6c009110 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4141 (unknown 0 0 4131983942) 0x7f2d84005190 con 0x7f2d6c009110 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d6c009110 legacy=0x7f2d6c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d81ffb640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d6c009110 legacy=0x7f2d6c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:13.621 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 12, 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 33787, 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 27, 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:13.622 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.623 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:13.624 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.625 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.626 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.627 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.629 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.631 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.632 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.633 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.634 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.634 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-04-25T21:40:13.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.634 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.634 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:13.634 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:13.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.203:6812/3164820986 conn(0x7f2d70036f00 legacy=0x7f2d700393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 10 monclient: shutdown 2024-04-25T21:40:13.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 >> v1:172.21.15.6:6790/0 conn(0x7f2d8c09ed30 legacy=0x7f2d8c1cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d91b29640 1 -- 172.21.15.6:0/3322737059 reap_dead start 2024-04-25T21:40:13.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 shutdown_connections 2024-04-25T21:40:13.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 >> 172.21.15.6:0/3322737059 conn(0x7f2d8c0a1c50 msgr2=0x7f2d8c099ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 shutdown_connections 2024-04-25T21:40:13.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.616+0000 7f2d9358c640 1 -- 172.21.15.6:0/3322737059 wait complete. 2024-04-25T21:40:13.637 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-25T21:40:13.637 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-25T21:40:13.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient: get_monmap_and_config 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient: monmap: 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:13.745903+0000 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:13.745903+0000 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:13.744 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:13.745 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 1 Processor -- start 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 1 -- start start 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient: init 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe006e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe009bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe013d0a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe009b0d0 con 0x7fdbe006e890 2024-04-25T21:40:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe00a4ca0 con 0x7fdbe013d0a0 2024-04-25T21:40:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe006f070 con 0x7fdbe009bab0 2024-04-25T21:40:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbe48ad640 10 monclient(hunting): authenticate will time out at 972.655592s 2024-04-25T21:40:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbde59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdbe006e890 0x7fdbe00a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60834/0 (socket says 172.21.15.6:60834) 2024-04-25T21:40:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbde59c640 1 -- 172.21.15.6:0/3336719082 learned_addr learned my addr 172.21.15.6:0/3336719082 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.740+0000 7fdbd5d9b640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fdbe009bab0 0x7fdbe00a4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33600/0 (socket says 172.21.15.6:33600) 2024-04-25T21:40:13.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbd5d9b640 1 -- 172.21.15.6:0/3336719082 >> v1:172.21.15.6:6790/0 conn(0x7fdbe009bab0 legacy=0x7fdbe00a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbde59c640 1 -- 172.21.15.6:0/3336719082 >> v1:172.21.15.6:6789/0 conn(0x7fdbe006e890 legacy=0x7fdbe00a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163756573 0 0) 0x7fdbe006f070 con 0x7fdbe009bab0 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): my global_id is 4418 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbcc003530 con 0x7fdbe009bab0 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbded9d640 1 -- 172.21.15.6:0/3336719082 >> v1:172.21.15.203:6789/0 conn(0x7fdbe013d0a0 legacy=0x7fdbe013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889360689 0 0) 0x7fdbd0001460 con 0x7fdbe006e890 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient(hunting): my global_id is 4434 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbe006f070 con 0x7fdbe006e890 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2302054012 0 0) 0x7fdbcc003530 con 0x7fdbe009bab0 2024-04-25T21:40:13.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 >> v1:172.21.15.203:6789/0 conn(0x7fdbe013d0a0 legacy=0x7fdbe013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 >> v1:172.21.15.6:6789/0 conn(0x7fdbe006e890 legacy=0x7fdbe00a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 monclient: found mon.noname-b 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe009adf0 con 0x7fdbe009bab0 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdbc8001990 con 0x7fdbe009bab0 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: dump: 2024-04-25T21:40:13.751 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 5 monclient: authenticate success, global_id 4418 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbc8001c80 con 0x7fdbe009bab0 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:13.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 1 -- 172.21.15.6:0/3336719082 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdbc8004990 con 0x7fdbe009bab0 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 4 set_mon_vals no callback set 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbddd9b640 10 monclient: dump: 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 10 monclient: shutdown 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 1 -- 172.21.15.6:0/3336719082 >> v1:172.21.15.6:6790/0 conn(0x7fdbe009bab0 legacy=0x7fdbe00a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 1 -- 172.21.15.6:0/3336719082 shutdown_connections 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 1 -- 172.21.15.6:0/3336719082 >> 172.21.15.6:0/3336719082 conn(0x7fdbe00a5f80 msgr2=0x7fdbe00a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 1 -- 172.21.15.6:0/3336719082 shutdown_connections 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 1 -- 172.21.15.6:0/3336719082 wait complete. 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.744+0000 7fdbe48ad640 10 monclient: monmap: 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.754 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:13.755 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.755 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:13.755 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:13.755 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 1 Processor -- start 2024-04-25T21:40:13.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 1 -- start start 2024-04-25T21:40:13.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient: init 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe006e890 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe009bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe013d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe009c490 con 0x7fdbe013d0a0 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe00a91d0 con 0x7fdbe006e890 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe0122210 con 0x7fdbe009bab0 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient(hunting): authenticate will time out at 972.660874s 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbded9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdbe013d0a0 0x7fdbe01d4770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60848/0 (socket says 172.21.15.6:60848) 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbded9d640 1 -- 172.21.15.6:0/1636617754 learned_addr learned my addr 172.21.15.6:0/1636617754 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbded9d640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.6:6789/0 conn(0x7fdbe013d0a0 legacy=0x7fdbe01d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd5d9b640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.6:6790/0 conn(0x7fdbe009bab0 legacy=0x7fdbe01d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4123484691 0 0) 0x7fdbe009c490 con 0x7fdbe013d0a0 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): my global_id is 4437 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc0003530 con 0x7fdbe013d0a0 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2465235450 0 0) 0x7fdbe0122210 con 0x7fdbe009bab0 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): my global_id is 4421 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbe009c490 con 0x7fdbe009bab0 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbde59c640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbe006e890 legacy=0x7fdbe01cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1997935753 0 0) 0x7fdbc0003530 con 0x7fdbe013d0a0 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbe0122210 con 0x7fdbe013d0a0 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdbd8001f00 con 0x7fdbe013d0a0 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): dump: 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1009676923 0 0) 0x7fdbe009c490 con 0x7fdbe009bab0 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbc0003530 con 0x7fdbe009bab0 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdbc8001950 con 0x7fdbe009bab0 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): dump: 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.763 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4058987992 0 0) 0x7fdbe00a91d0 con 0x7fdbe006e890 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient(hunting): my global_id is 4372 2024-04-25T21:40:13.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbe009c490 con 0x7fdbe006e890 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4075101969 0 0) 0x7fdbe0122210 con 0x7fdbe013d0a0 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.6:6790/0 conn(0x7fdbe009bab0 legacy=0x7fdbe01d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbe006e890 legacy=0x7fdbe01cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 monclient: found mon.a 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe0124220 con 0x7fdbe013d0a0 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 5 monclient: authenticate success, global_id 4437 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient: _renew_subs 2024-04-25T21:40:13.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe0122af0 con 0x7fdbe013d0a0 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient: _renew_subs 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.748+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdbe0124af0 con 0x7fdbe013d0a0 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbd8003cd0 con 0x7fdbe013d0a0 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdbd8004020 con 0x7fdbe013d0a0 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbdd59a640 4 set_mon_vals no callback set 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient: dump: 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.766 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbe48ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdbc40065d0 con 0x7fdbc4002870 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fdbd80041c0 con 0x7fdbe013d0a0 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3898+0+0 (unknown 2549805797 0 0) 0x7fdbd8044830 con 0x7fdbe013d0a0 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd5d9b640 10 monclient: get_auth_request con 0x7fdbc0036cc0 auth_method 0 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbde59c640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbc4002870 legacy=0x7fdbc4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1123065650 0 0) 0x7fdbc40065d0 con 0x7fdbc4002870 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient(hunting): my global_id is 4375 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbe0122210 con 0x7fdbc4002870 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2765366836 0 0) 0x7fdbe0122210 con 0x7fdbc4002870 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbc40065d0 con 0x7fdbc4002870 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdbd0001a40 con 0x7fdbc4002870 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.752+0000 7fdbd77fe640 10 monclient: dump: 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.768 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.756+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 213300347 0 0) 0x7fdbc40065d0 con 0x7fdbc4002870 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.756+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdbc400a160 con 0x7fdbc4002870 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.756+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdbd00019a0 con 0x7fdbc4002870 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.756+0000 7fdbd77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.756+0000 7fdbd77fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.756+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbc4002870 legacy=0x7fdbc4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.756+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbc4002870 legacy=0x7fdbc4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.760+0000 7fdbe48ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.760+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdbc4009e50 con 0x7fdbc4009110 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbde59c640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbc4009110 legacy=0x7fdbc40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4000967492 0 0) 0x7fdbc4009e50 con 0x7fdbc4009110 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient(hunting): my global_id is 4378 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbc40065d0 con 0x7fdbc4009110 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3254653152 0 0) 0x7fdbc40065d0 con 0x7fdbc4009110 2024-04-25T21:40:13.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdbc4009e50 con 0x7fdbc4009110 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fdbd0004b80 con 0x7fdbc4009110 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient: dump: 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.772 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1455717352 0 0) 0x7fdbc4009e50 con 0x7fdbc4009110 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdbc400c3e0 con 0x7fdbc4009110 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4311 (unknown 0 0 2135230376) 0x7fdbd0004e20 con 0x7fdbc4009110 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbc4009110 legacy=0x7fdbc40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.764+0000 7fdbd77fe640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6789/0 conn(0x7fdbc4009110 legacy=0x7fdbc40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:13.773 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 12, 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 33895, 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 27, 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.774 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.775 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:13.776 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:13.777 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:13.778 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:13.779 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.780 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:13.781 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:13.782 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:13.783 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.784 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.785 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 6 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:13.786 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:13.787 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:13.787 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:13.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:13.787 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:13.787 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.203:6812/3164820986 conn(0x7fdbc0036cc0 legacy=0x7fdbc0039180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 10 monclient: shutdown 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 >> v1:172.21.15.6:6789/0 conn(0x7fdbe013d0a0 legacy=0x7fdbe01d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbded9d640 1 -- 172.21.15.6:0/1636617754 reap_dead start 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbded9d640 1 -- 172.21.15.6:0/1636617754 reap_dead start 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 shutdown_connections 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 >> 172.21.15.6:0/1636617754 conn(0x7fdbe00a5f80 msgr2=0x7fdbe009f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 shutdown_connections 2024-04-25T21:40:13.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.768+0000 7fdbe48ad640 1 -- 172.21.15.6:0/1636617754 wait complete. 2024-04-25T21:40:13.790 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-25T21:40:13.790 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-25T21:40:13.790 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-25T21:40:13.790 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-25T21:40:13.790 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-25T21:40:13.791 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T21:40:13.791 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:40:13.812 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T21:40:13.813 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-25T21:40:13.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-25T21:40:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient: get_monmap_and_config 2024-04-25T21:40:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient: monmap: 2024-04-25T21:40:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.921 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:13.921182+0000 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:13.921182+0000 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 1 Processor -- start 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 1 -- start start 2024-04-25T21:40:13.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient: init 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient(hunting): picked mon.noname-b con 0x7f6a3c09b690 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient(hunting): picked mon.noname-a con 0x7f6a3c09ba70 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient(hunting): picked mon.noname-c con 0x7f6a3c09beb0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c099bc0 con 0x7f6a3c09ba70 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c0998e0 con 0x7f6a3c09beb0 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c0a0dc0 con 0x7f6a3c09b690 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a42721640 10 monclient(hunting): authenticate will time out at 972.830622s 2024-04-25T21:40:13.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a3b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a3c09ba70 0x7f6a3c13bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60860/0 (socket says 172.21.15.6:60860) 2024-04-25T21:40:13.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a3b7fe640 1 -- 172.21.15.6:0/2567933719 learned_addr learned my addr 172.21.15.6:0/2567933719 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.916+0000 7f6a40cbe640 1 -- 172.21.15.6:0/2567933719 >> v1:172.21.15.203:6789/0 conn(0x7f6a3c09beb0 legacy=0x7f6a3c13f3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3b7fe640 1 -- 172.21.15.6:0/2567933719 >> v1:172.21.15.6:6789/0 conn(0x7f6a3c09ba70 legacy=0x7f6a3c13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3bfff640 1 -- 172.21.15.6:0/2567933719 >> v1:172.21.15.6:6790/0 conn(0x7f6a3c09b690 legacy=0x7f6a3c136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050383179 0 0) 0x7f6a3c099bc0 con 0x7f6a3c09ba70 2024-04-25T21:40:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): my global_id is 4440 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a24003530 con 0x7f6a3c09ba70 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799371923 0 0) 0x7f6a3c0a0dc0 con 0x7f6a3c09b690 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient(hunting): my global_id is 4424 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a3c099bc0 con 0x7f6a3c09b690 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3662193132 0 0) 0x7f6a3c099bc0 con 0x7f6a3c09b690 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 >> v1:172.21.15.203:6789/0 conn(0x7f6a3c09beb0 legacy=0x7f6a3c13f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 >> v1:172.21.15.6:6789/0 conn(0x7f6a3c09ba70 legacy=0x7f6a3c13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 monclient: found mon.noname-b 2024-04-25T21:40:13.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c143b20 con 0x7f6a3c09b690 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6a30001920 con 0x7f6a3c09b690 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: dump: 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.932 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 5 monclient: authenticate success, global_id 4424 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a30001c70 con 0x7f6a3c09b690 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 1 -- 172.21.15.6:0/2567933719 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6a300037c0 con 0x7f6a3c09b690 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 4 set_mon_vals no callback set 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3affd640 10 monclient: dump: 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.934 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient: shutdown 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- 172.21.15.6:0/2567933719 >> v1:172.21.15.6:6790/0 conn(0x7f6a3c09b690 legacy=0x7f6a3c136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- 172.21.15.6:0/2567933719 shutdown_connections 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- 172.21.15.6:0/2567933719 >> 172.21.15.6:0/2567933719 conn(0x7f6a3c0a1c50 msgr2=0x7f6a3c0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- 172.21.15.6:0/2567933719 shutdown_connections 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- 172.21.15.6:0/2567933719 wait complete. 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient: build_initial_monmap 2024-04-25T21:40:13.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient: monmap: 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 Processor -- start 2024-04-25T21:40:13.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- start start 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient: init 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient(hunting): picked mon.noname-a con 0x7f6a3c09b690 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient(hunting): picked mon.noname-b con 0x7f6a3c09ba70 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient(hunting): picked mon.noname-c con 0x7f6a3c09beb0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c0a13b0 con 0x7f6a3c09b690 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c09ece0 con 0x7f6a3c09ba70 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c12b220 con 0x7f6a3c09beb0 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a42721640 10 monclient(hunting): authenticate will time out at 972.835530s 2024-04-25T21:40:13.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a40cbe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6a3c09beb0 0x7f6a3c1d43a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33622/0 (socket says 172.21.15.6:33622) 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3bfff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a3c09b690 0x7f6a3c1cbc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60866/0 (socket says 172.21.15.6:60866) 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a40cbe640 1 -- 172.21.15.6:0/2010003451 learned_addr learned my addr 172.21.15.6:0/2010003451 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.920+0000 7f6a3b7fe640 1 -- 172.21.15.6:0/2010003451 >> v1:172.21.15.203:6789/0 conn(0x7f6a3c09ba70 legacy=0x7f6a3c1d0c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a40cbe640 1 -- 172.21.15.6:0/2010003451 >> v1:172.21.15.6:6790/0 conn(0x7f6a3c09beb0 legacy=0x7f6a3c1d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a3bfff640 1 -- 172.21.15.6:0/2010003451 >> v1:172.21.15.6:6789/0 conn(0x7f6a3c09b690 legacy=0x7f6a3c1cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668738228 0 0) 0x7f6a3c12b220 con 0x7f6a3c09beb0 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): my global_id is 4427 2024-04-25T21:40:13.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a28003530 con 0x7f6a3c09beb0 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2230062304 0 0) 0x7f6a3c143b20 con 0x7f6a3c09b690 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): my global_id is 4443 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a3c12b220 con 0x7f6a3c09b690 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3130933353 0 0) 0x7f6a28003530 con 0x7f6a3c09beb0 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a40cbe640 1 --1- 172.21.15.6:0/2010003451 >> v1:172.21.15.6:6790/0 conn(0x7f6a3c09beb0 0x7f6a3c1d43a0 :-1 s=OPENED pgs=155 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f6a340047c0 mon_map magic: 0 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a3c143b20 con 0x7f6a3c09beb0 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2681616387 0 0) 0x7f6a3c12b220 con 0x7f6a3c09b690 2024-04-25T21:40:13.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a28003530 con 0x7f6a3c09b690 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6a300037c0 con 0x7f6a3c09b690 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.940 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6a340047c0 con 0x7f6a3c09beb0 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.941 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1878491460 0 0) 0x7f6a28003530 con 0x7f6a3c09b690 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 >> v1:172.21.15.6:6790/0 conn(0x7f6a3c09beb0 legacy=0x7f6a3c1d43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 >> v1:172.21.15.203:6789/0 conn(0x7f6a3c09ba70 legacy=0x7f6a3c1d0c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 monclient: found mon.a 2024-04-25T21:40:13.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c12c4a0 con 0x7f6a3c09b690 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 5 monclient: authenticate success, global_id 4443 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 10 monclient: _renew_subs 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3059836009 0 0) 0x7f6a3c143b20 con 0x7f6a3c09beb0 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c131080 con 0x7f6a3c09b690 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 10 monclient: _renew_subs 2024-04-25T21:40:13.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a3c09e5b0 con 0x7f6a3c09b690 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a30001bf0 con 0x7f6a3c09b690 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6a30003f50 con 0x7f6a3c09b690 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a3a7fc640 4 set_mon_vals no callback set 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 10 monclient: dump: 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f6a300040f0 con 0x7f6a3c09b690 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:13.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a1c005090 con 0x7f6a3c09b690 2024-04-25T21:40:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3898+0+0 (unknown 2549805797 0 0) 0x7f6a300444e0 con 0x7f6a3c09b690 2024-04-25T21:40:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.924+0000 7f6a3b7fe640 10 monclient: get_auth_request con 0x7f6a28036a10 auth_method 0 2024-04-25T21:40:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.932+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f6a3c0a13b0 con 0x7f6a3c09b690 2024-04-25T21:40:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.932+0000 7f6a38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:13.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:13.932+0000 7f6a38ff9640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.140+0000 7f6a42721640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-25T21:40:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.140+0000 7f6a42721640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.140+0000 7f6a42721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:14.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.140+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6a1c005380 con 0x7f6a3c09b690 2024-04-25T21:40:14.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a38ff9640 1 -- 172.21.15.6:0/2010003451 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1401 (unknown 254746657 0 888435006) 0x7f6a30015740 con 0x7f6a3c09b690 2024-04-25T21:40:14.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:14.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a38ff9640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:14.146 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","modified":"2024-04-25T21:39:14.743126Z","created":"2024-04-25T21:39:14.743126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 >> v1:172.21.15.203:6812/3164820986 conn(0x7f6a28036a10 legacy=0x7f6a28038ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 10 monclient: shutdown 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 >> v1:172.21.15.6:6789/0 conn(0x7f6a3c09b690 legacy=0x7f6a3c1cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 shutdown_connections 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 >> 172.21.15.6:0/2010003451 conn(0x7f6a3c0a1c50 msgr2=0x7f6a3c09ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 shutdown_connections 2024-04-25T21:40:14.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:14.144+0000 7f6a42721640 1 -- 172.21.15.6:0/2010003451 wait complete. 2024-04-25T21:40:17.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-25T21:40:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient: get_monmap_and_config 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient: build_initial_monmap 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient: monmap: 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:17.297859+0000 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:17.297859+0000 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:17.297 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 1 Processor -- start 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 1 -- start start 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient: init 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient(hunting): picked mon.noname-b con 0x7f83b009df10 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient(hunting): picked mon.noname-a con 0x7f83b009cb10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient(hunting): picked mon.noname-c con 0x7f83b01399f0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b009bc10 con 0x7f83b009cb10 2024-04-25T21:40:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b0073230 con 0x7f83b01399f0 2024-04-25T21:40:17.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b009f2f0 con 0x7f83b009df10 2024-04-25T21:40:17.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:17.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b81ea640 10 monclient(hunting): authenticate will time out at 976.207770s 2024-04-25T21:40:17.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b5f86640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f83b009df10 0x7f83b009e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33628/0 (socket says 172.21.15.6:33628) 2024-04-25T21:40:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b5f86640 1 -- 172.21.15.6:0/817608758 learned_addr learned my addr 172.21.15.6:0/817608758 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.292+0000 7f83b6787640 1 -- 172.21.15.6:0/817608758 >> v1:172.21.15.203:6789/0 conn(0x7f83b01399f0 legacy=0x7f83b013bde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b5785640 1 -- 172.21.15.6:0/817608758 >> v1:172.21.15.6:6789/0 conn(0x7f83b009cb10 legacy=0x7f83b01382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b5f86640 1 -- 172.21.15.6:0/817608758 >> v1:172.21.15.6:6790/0 conn(0x7f83b009df10 legacy=0x7f83b009e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1620525969 0 0) 0x7f83b009bc10 con 0x7f83b009cb10 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient(hunting): _negotiate 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient(hunting): my global_id is 4446 2024-04-25T21:40:17.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f839c003530 con 0x7f83b009cb10 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 568389767 0 0) 0x7f839c003530 con 0x7f83b009cb10 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 >> v1:172.21.15.6:6790/0 conn(0x7f83b009df10 legacy=0x7f83b009e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 >> v1:172.21.15.203:6789/0 conn(0x7f83b01399f0 legacy=0x7f83b013bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: _finish_hunting 0 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 monclient: found mon.noname-a 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b009b930 con 0x7f83b009cb10 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: _finish_auth 0 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: _check_auth_tickets 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f83a0001920 con 0x7f83b009cb10 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 5 monclient: authenticate success, global_id 4446 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:17.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: dump: 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f83a0001c70 con 0x7f83b009cb10 2024-04-25T21:40:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 1 -- 172.21.15.6:0/817608758 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f83a00037c0 con 0x7f83b009cb10 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b4f84640 10 monclient: dump: 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:17.310 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 4 set_mon_vals no callback set 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 10 monclient: shutdown 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 1 -- 172.21.15.6:0/817608758 >> v1:172.21.15.6:6789/0 conn(0x7f83b009cb10 legacy=0x7f83b01382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 1 -- 172.21.15.6:0/817608758 shutdown_connections 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 1 -- 172.21.15.6:0/817608758 >> 172.21.15.6:0/817608758 conn(0x7f83b00a1c50 msgr2=0x7f83b00a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 1 -- 172.21.15.6:0/817608758 shutdown_connections 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.296+0000 7f83b81ea640 1 -- 172.21.15.6:0/817608758 wait complete. 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient: build_initial_monmap 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient: monmap: 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:17.311 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 1 Processor -- start 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 1 -- start start 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient: init 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient(hunting): picked mon.noname-a con 0x7f83b009df10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient(hunting): picked mon.noname-c con 0x7f83b01399f0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient(hunting): picked mon.noname-b con 0x7f83b0138180 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:17.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b006ec80 con 0x7f83b009df10 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b006e9a0 con 0x7f83b0138180 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b0072a70 con 0x7f83b01399f0 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient(hunting): authenticate will time out at 976.212957s 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b5f86640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f83b009df10 0x7f83b0137360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60886/0 (socket says 172.21.15.6:60886) 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b5f86640 1 -- 172.21.15.6:0/944026179 learned_addr learned my addr 172.21.15.6:0/944026179 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b5785640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f83b01399f0 0x7f83b0137a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:33632/0 (socket says 172.21.15.6:33632) 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b6787640 1 -- 172.21.15.6:0/944026179 >> v1:172.21.15.203:6789/0 conn(0x7f83b0138180 legacy=0x7f83b01cfbc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b5785640 1 -- 172.21.15.6:0/944026179 >> v1:172.21.15.6:6790/0 conn(0x7f83b01399f0 legacy=0x7f83b0137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b5f86640 1 -- 172.21.15.6:0/944026179 >> v1:172.21.15.6:6789/0 conn(0x7f83b009df10 legacy=0x7f83b0137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 671815874 0 0) 0x7f83b006ec80 con 0x7f83b009df10 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:17.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): my global_id is 4449 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8394003530 con 0x7f83b009df10 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3313822028 0 0) 0x7f8394003530 con 0x7f83b009df10 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83b006ec80 con 0x7f83b009df10 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f83ac001b70 con 0x7f83b009df10 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): dump: 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:17.314 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 278932240 0 0) 0x7f83b006ec80 con 0x7f83b009df10 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 >> v1:172.21.15.6:6790/0 conn(0x7f83b01399f0 legacy=0x7f83b0137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 >> v1:172.21.15.203:6789/0 conn(0x7f83b0138180 legacy=0x7f83b01cfbc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 monclient: found mon.a 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b006f660 con 0x7f83b009df10 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:17.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 5 monclient: authenticate success, global_id 4449 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient: _renew_subs 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.300+0000 7f83b81ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f83b01d12e0 con 0x7f83b009df10 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 10 monclient: _renew_subs 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83b01d1680 con 0x7f83b009df10 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f83ac004aa0 con 0x7f83b009df10 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f83ac004df0 con 0x7f83b009df10 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 10 monclient: dump: 2024-04-25T21:40:17.316 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a7fff640 4 set_mon_vals no callback set 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f83ac004fd0 con 0x7f83b009df10 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8398005090 con 0x7f83b009df10 2024-04-25T21:40:17.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3898+0+0 (unknown 2549805797 0 0) 0x7f83ac044650 con 0x7f83b009df10 2024-04-25T21:40:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.304+0000 7f83b5785640 10 monclient: get_auth_request con 0x7f8394036a40 auth_method 0 2024-04-25T21:40:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.308+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f83ac015850 con 0x7f83b009df10 2024-04-25T21:40:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.308+0000 7f83a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:17.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.308+0000 7f83a67fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-25T21:40:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8398005380 con 0x7f83b009df10 2024-04-25T21:40:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83a67fc640 1 -- 172.21.15.6:0/944026179 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1401 (unknown 254746657 0 1383706469) 0x7f83ac015150 con 0x7f83b009df10 2024-04-25T21:40:17.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:17.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83a67fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:17.528 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","modified":"2024-04-25T21:39:14.743126Z","created":"2024-04-25T21:39:14.743126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-25T21:40:17.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 >> v1:172.21.15.203:6812/3164820986 conn(0x7f8394036a40 legacy=0x7f8394038f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:17.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 10 monclient: shutdown 2024-04-25T21:40:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.524+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 >> v1:172.21.15.6:6789/0 conn(0x7f83b009df10 legacy=0x7f83b0137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.528+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 shutdown_connections 2024-04-25T21:40:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.528+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 >> 172.21.15.6:0/944026179 conn(0x7f83b00a1c50 msgr2=0x7f83b00a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.528+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 shutdown_connections 2024-04-25T21:40:17.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:17.528+0000 7f83b81ea640 1 -- 172.21.15.6:0/944026179 wait complete. 2024-04-25T21:40:20.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-25T21:40:20.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient: get_monmap_and_config 2024-04-25T21:40:20.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient: build_initial_monmap 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient: monmap: 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:20.677588+0000 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:20.677588+0000 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:20.676 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:20.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 1 Processor -- start 2024-04-25T21:40:20.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 1 -- start start 2024-04-25T21:40:20.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient: init 2024-04-25T21:40:20.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:20.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient(hunting): picked mon.noname-a con 0x7f35e009b690 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient(hunting): picked mon.noname-c con 0x7f35e009ba70 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient(hunting): picked mon.noname-b con 0x7f35e009beb0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e0099bc0 con 0x7f35e009b690 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e00998e0 con 0x7f35e009ba70 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e00a0dc0 con 0x7f35e009beb0 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35e539a640 10 monclient(hunting): authenticate will time out at 979.587417s 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35deffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f35e009b690 0x7f35e0136e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60898/0 (socket says 172.21.15.6:60898) 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 learned_addr learned my addr 172.21.15.6:0/794168040 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:20.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.672+0000 7f35de7fc640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35e009ba70 legacy=0x7f35e013bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:20.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.676+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6789/0 conn(0x7f35e009b690 legacy=0x7f35e0136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:20.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.676+0000 7f35df7fe640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6790/0 conn(0x7f35e009beb0 legacy=0x7f35e013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:20.876+0000 7f35de7fc640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35e009ba70 legacy=0x7f35e013bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:21.280+0000 7f35de7fc640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35e009ba70 legacy=0x7f35e013bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:22.084+0000 7f35de7fc640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35e009ba70 legacy=0x7f35e013bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:23.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient(hunting): tick 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient(hunting): _check_auth_tickets 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 1 monclient(hunting): continuing hunt 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6790/0 conn(0x7f35e009beb0 legacy=0x7f35e013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35e009ba70 legacy=0x7f35e013bca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6789/0 conn(0x7f35e009b690 legacy=0x7f35e0136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f35c00037d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f35c0007490 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f35e009b690 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e0143b20 con 0x7f35e009b690 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e00998e0 con 0x7f35c00037d0 2024-04-25T21:40:23.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c00109e0 con 0x7f35c0007490 2024-04-25T21:40:23.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.672+0000 7f35dcff9640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:23.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.676+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c00037d0 legacy=0x7f35c0005c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.676+0000 7f35df7fe640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6789/0 conn(0x7f35e009b690 legacy=0x7f35c000d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:23.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.676+0000 7f35de7fc640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6790/0 conn(0x7f35c0007490 legacy=0x7f35c0009920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:23.880+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c00037d0 legacy=0x7f35c0005c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:24.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:24.284+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c00037d0 legacy=0x7f35c0005c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:25.088+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c00037d0 legacy=0x7f35c0005c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient(hunting): tick 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient(hunting): _check_auth_tickets 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 1 monclient(hunting): continuing hunt 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6790/0 conn(0x7f35c0007490 legacy=0x7f35c0009920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c00037d0 legacy=0x7f35c0005c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6789/0 conn(0x7f35e009b690 legacy=0x7f35c000d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient(hunting): picked mon.noname-c con 0x7f35c0010d30 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient(hunting): picked mon.noname-a con 0x7f35c00063f0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient(hunting): picked mon.noname-b con 0x7f35c0007490 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c0011c90 con 0x7f35c00063f0 2024-04-25T21:40:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e00998e0 con 0x7f35c0010d30 2024-04-25T21:40:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c00067d0 con 0x7f35c0007490 2024-04-25T21:40:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35dcff9640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c0010d30 legacy=0x7f35c0011170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35de7fc640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6789/0 conn(0x7f35c00063f0 legacy=0x7f35c0005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.676+0000 7f35df7fe640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6790/0 conn(0x7f35c0007490 legacy=0x7f35c000d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:26.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:26.880+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c0010d30 legacy=0x7f35c0011170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:27.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:27.284+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c0010d30 legacy=0x7f35c0011170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:28.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.088+0000 7f35deffd640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c0010d30 legacy=0x7f35c0011170 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 279960313 0 0) 0x7f35c0011c90 con 0x7f35c00063f0 2024-04-25T21:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:28.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4452 2024-04-25T21:40:28.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35cc003530 con 0x7f35c00063f0 2024-04-25T21:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1830424913 0 0) 0x7f35c00067d0 con 0x7f35c0007490 2024-04-25T21:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:28.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 10 monclient(hunting): my global_id is 4430 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.652+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35c0011c90 con 0x7f35c0007490 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3965485288 0 0) 0x7f35cc003530 con 0x7f35c00063f0 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6790/0 conn(0x7f35c0007490 legacy=0x7f35c000d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.203:6789/0 conn(0x7f35c0010d30 legacy=0x7f35c0011170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 monclient: found mon.noname-a 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c0009930 con 0x7f35c00063f0 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:28.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f35c80043f0 con 0x7f35c00063f0 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 5 monclient: authenticate success, global_id 4452 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: dump: 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35de7fc640 1 --1- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6789/0 conn(0x7f35c00063f0 0x7f35c0005c20 :-1 s=OPENED pgs=171 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f35c8004780 config(0 keys) 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35c8004780 con 0x7f35c00063f0 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:28.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 4 set_mon_vals no callback set 2024-04-25T21:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 10 monclient: shutdown 2024-04-25T21:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 1 -- 172.21.15.6:0/794168040 >> v1:172.21.15.6:6789/0 conn(0x7f35c00063f0 legacy=0x7f35c0005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 1 -- 172.21.15.6:0/794168040 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f35c8004ad0 con 0x7f35c00063f0 2024-04-25T21:40:28.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35ddffb640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 1 -- 172.21.15.6:0/794168040 shutdown_connections 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 1 -- 172.21.15.6:0/794168040 >> 172.21.15.6:0/794168040 conn(0x7f35e00a1c50 msgr2=0x7f35e00a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 1 -- 172.21.15.6:0/794168040 shutdown_connections 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 1 -- 172.21.15.6:0/794168040 wait complete. 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 10 monclient: build_initial_monmap 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.656+0000 7f35e539a640 10 monclient: monmap: 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:28.663 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 1 Processor -- start 2024-04-25T21:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 1 -- start start 2024-04-25T21:40:28.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient: init 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient(hunting): picked mon.noname-a con 0x7f35c00063f0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient(hunting): picked mon.noname-b con 0x7f35c0007490 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient(hunting): picked mon.noname-c con 0x7f35c0010d30 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e012ccf0 con 0x7f35c00063f0 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e00a13b0 con 0x7f35c0007490 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e009ac90 con 0x7f35c0010d30 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:28.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35e539a640 10 monclient(hunting): authenticate will time out at 987.573879s 2024-04-25T21:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35de7fc640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f35c0007490 legacy=0x7f35e01d0f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35df7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f35c0010d30 0x7f35e01d4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35988/0 (socket says 172.21.15.6:35988) 2024-04-25T21:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.660+0000 7f35df7fe640 1 -- 172.21.15.6:0/2641377819 learned_addr learned my addr 172.21.15.6:0/2641377819 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:28.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35deffd640 1 -- 172.21.15.6:0/2641377819 >> v1:172.21.15.6:6789/0 conn(0x7f35c00063f0 legacy=0x7f35e0130740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35df7fe640 1 -- 172.21.15.6:0/2641377819 >> v1:172.21.15.6:6790/0 conn(0x7f35c0010d30 legacy=0x7f35e01d4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823288064 0 0) 0x7f35e012ccf0 con 0x7f35c00063f0 2024-04-25T21:40:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): my global_id is 4455 2024-04-25T21:40:28.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35c0003500 con 0x7f35c00063f0 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2492216911 0 0) 0x7f35c0003500 con 0x7f35c00063f0 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35e012ccf0 con 0x7f35c00063f0 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f35d8001ef0 con 0x7f35c00063f0 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): dump: 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 47470110 0 0) 0x7f35e012ccf0 con 0x7f35c00063f0 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 >> v1:172.21.15.6:6790/0 conn(0x7f35c0010d30 legacy=0x7f35e01d4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 >> v1:172.21.15.203:6789/0 conn(0x7f35c0007490 legacy=0x7f35e01d0f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 monclient: found mon.a 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:28.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e012efe0 con 0x7f35c00063f0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35e539a640 5 monclient: authenticate success, global_id 4455 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35e539a640 10 monclient: _renew_subs 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35e539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35e01c8c60 con 0x7f35c00063f0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35e539a640 10 monclient: _renew_subs 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35e539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f35e0131020 con 0x7f35c00063f0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35d8004e20 con 0x7f35c00063f0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f35d8005170 con 0x7f35c00063f0 2024-04-25T21:40:28.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35d77fe640 10 monclient: dump: 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:28.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.664+0000 7f35dd7fa640 4 set_mon_vals no callback set 2024-04-25T21:40:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.668+0000 7f35e539a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.668+0000 7f35e539a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.668+0000 7f35e539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.668+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f35d8005310 con 0x7f35c00063f0 2024-04-25T21:40:28.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.668+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c4005090 con 0x7f35c00063f0 2024-04-25T21:40:28.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.668+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3898+0+0 (unknown 2549805797 0 0) 0x7f35d8044850 con 0x7f35c00063f0 2024-04-25T21:40:28.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.668+0000 7f35de7fc640 10 monclient: get_auth_request con 0x7f35c0038570 auth_method 0 2024-04-25T21:40:28.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.672+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f35d8015a50 con 0x7f35c00063f0 2024-04-25T21:40:28.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.672+0000 7f35d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:28.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.672+0000 7f35d77fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:28.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.900+0000 7f35e539a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-25T21:40:28.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.900+0000 7f35e539a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:28.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.900+0000 7f35e539a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.900+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f35c4005380 con 0x7f35c00063f0 2024-04-25T21:40:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.900+0000 7f35d77fe640 1 -- 172.21.15.6:0/2641377819 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1394 (unknown 254746657 0 749906765) 0x7f35d8015350 con 0x7f35c00063f0 2024-04-25T21:40:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.900+0000 7f35d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:28.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.900+0000 7f35d77fe640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:28.904 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","modified":"2024-04-25T21:39:14.743126Z","created":"2024-04-25T21:39:14.743126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-25T21:40:28.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 >> v1:172.21.15.203:6812/3164820986 conn(0x7f35c0038570 legacy=0x7f35c003aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 10 monclient: shutdown 2024-04-25T21:40:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 >> v1:172.21.15.6:6789/0 conn(0x7f35c00063f0 legacy=0x7f35e0130740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 shutdown_connections 2024-04-25T21:40:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 >> 172.21.15.6:0/2641377819 conn(0x7f35e00a1c50 msgr2=0x7f35e01343e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 shutdown_connections 2024-04-25T21:40:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:28.904+0000 7f35e539a640 1 -- 172.21.15.6:0/2641377819 wait complete. 2024-04-25T21:40:28.923 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-25T21:40:28.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-25T21:40:28.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-25T21:40:29.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient: get_monmap_and_config 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient: build_initial_monmap 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient: monmap: 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:29.043738+0000 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:29.043738+0000 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:29.042 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 1 Processor -- start 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 1 -- start start 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient: init 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient(hunting): picked mon.noname-c con 0x7f70fc0a0b60 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient(hunting): picked mon.noname-a con 0x7f70fc09ed30 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient(hunting): picked mon.noname-b con 0x7f70fc09f180 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc09aa70 con 0x7f70fc09ed30 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc072c40 con 0x7f70fc0a0b60 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc09bbf0 con 0x7f70fc09f180 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7102907640 10 monclient(hunting): authenticate will time out at 987.953500s 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7100ea4640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f70fc09f180 0x7f70fc13c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:35994/0 (socket says 172.21.15.6:35994) 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f7100ea4640 1 -- 172.21.15.6:0/1596200240 learned_addr learned my addr 172.21.15.6:0/1596200240 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:29.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.040+0000 7f70fbfff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f70fc0a0b60 legacy=0x7f70fc09a360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:29.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70fb7fe640 1 -- 172.21.15.6:0/1596200240 >> v1:172.21.15.6:6789/0 conn(0x7f70fc09ed30 legacy=0x7f70fc138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7100ea4640 1 -- 172.21.15.6:0/1596200240 >> v1:172.21.15.6:6790/0 conn(0x7f70fc09f180 legacy=0x7f70fc13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3124207830 0 0) 0x7f70fc09aa70 con 0x7f70fc09ed30 2024-04-25T21:40:29.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): my global_id is 4458 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70e4003530 con 0x7f70fc09ed30 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756374356 0 0) 0x7f70fc09bbf0 con 0x7f70fc09f180 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient(hunting): my global_id is 4433 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70fc09aa70 con 0x7f70fc09f180 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2582521153 0 0) 0x7f70e4003530 con 0x7f70fc09ed30 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 >> v1:172.21.15.6:6790/0 conn(0x7f70fc09f180 legacy=0x7f70fc13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 >> v1:172.21.15.203:6789/0 conn(0x7f70fc0a0b60 legacy=0x7f70fc09a360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 monclient: found mon.noname-a 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70fc09bed0 con 0x7f70fc09ed30 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f70f00019e0 con 0x7f70fc09ed30 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 5 monclient: authenticate success, global_id 4458 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: dump: 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.049 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70f0001cd0 con 0x7f70fc09ed30 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 1 -- 172.21.15.6:0/1596200240 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f70f00037c0 con 0x7f70fc09ed30 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70faffd640 10 monclient: dump: 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 4 set_mon_vals no callback set 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient: shutdown 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:29.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- 172.21.15.6:0/1596200240 >> v1:172.21.15.6:6789/0 conn(0x7f70fc09ed30 legacy=0x7f70fc138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- 172.21.15.6:0/1596200240 shutdown_connections 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- 172.21.15.6:0/1596200240 >> 172.21.15.6:0/1596200240 conn(0x7f70fc0a1c50 msgr2=0x7f70fc0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- 172.21.15.6:0/1596200240 shutdown_connections 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- 172.21.15.6:0/1596200240 wait complete. 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient: build_initial_monmap 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient: monmap: 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:29.052 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:29.053 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 Processor -- start 2024-04-25T21:40:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- start start 2024-04-25T21:40:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient: init 2024-04-25T21:40:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient(hunting): picked mon.noname-c con 0x7f70fc09ed30 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient(hunting): picked mon.noname-b con 0x7f70fc09f180 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient(hunting): picked mon.noname-a con 0x7f70fc0a0b60 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc091870 con 0x7f70fc0a0b60 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc09e5b0 con 0x7f70fc09f180 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc090400 con 0x7f70fc09ed30 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f7102907640 10 monclient(hunting): authenticate will time out at 987.959263s 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70fbfff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f70fc09ed30 0x7f70fc1cbb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:36008/0 (socket says 172.21.15.6:36008) 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70fbfff640 1 -- 172.21.15.6:0/213368471 learned_addr learned my addr 172.21.15.6:0/213368471 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.044+0000 7f70fb7fe640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.203:6789/0 conn(0x7f70fc09f180 legacy=0x7f70fc1d0b00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:29.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7100ea4640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70fc0a0b60 legacy=0x7f70fc1d4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70fbfff640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6790/0 conn(0x7f70fc09ed30 legacy=0x7f70fc1cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948746845 0 0) 0x7f70fc091870 con 0x7f70fc0a0b60 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): my global_id is 4461 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70e8003530 con 0x7f70fc0a0b60 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 762578742 0 0) 0x7f70fc090400 con 0x7f70fc09ed30 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): my global_id is 4436 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70fc091870 con 0x7f70fc09ed30 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3009628436 0 0) 0x7f70e8003530 con 0x7f70fc0a0b60 2024-04-25T21:40:29.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70fc090400 con 0x7f70fc0a0b60 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f70f4001e40 con 0x7f70fc0a0b60 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4019833482 0 0) 0x7f70fc091870 con 0x7f70fc09ed30 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70e8003530 con 0x7f70fc09ed30 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f70ec004340 con 0x7f70fc09ed30 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 814289992 0 0) 0x7f70fc090400 con 0x7f70fc0a0b60 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6790/0 conn(0x7f70fc09ed30 legacy=0x7f70fc1cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.203:6789/0 conn(0x7f70fc09f180 legacy=0x7f70fc1d0b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 monclient: found mon.a 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70fc091590 con 0x7f70fc0a0b60 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:29.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 5 monclient: authenticate success, global_id 4461 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 10 monclient: _renew_subs 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f70fc0924c0 con 0x7f70fc0a0b60 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 10 monclient: _renew_subs 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f70fc09dcb0 con 0x7f70fc0a0b60 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f70f4004e20 con 0x7f70fc0a0b60 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f70f4005170 con 0x7f70fc0a0b60 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70fa7fc640 4 set_mon_vals no callback set 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 10 monclient: dump: 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.063 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f70dc0065d0 con 0x7f70dc002870 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f70f4005310 con 0x7f70fc0a0b60 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3898+0+0 (unknown 2549805797 0 0) 0x7f70f4044770 con 0x7f70fc0a0b60 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.048+0000 7f70fb7fe640 10 monclient: get_auth_request con 0x7f70e8036b60 auth_method 0 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70fbfff640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70dc002870 legacy=0x7f70dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666470075 0 0) 0x7f70dc0065d0 con 0x7f70dc002870 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient(hunting): my global_id is 4464 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70fc090400 con 0x7f70dc002870 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 497542967 0 0) 0x7f70fc090400 con 0x7f70dc002870 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70dc0065d0 con 0x7f70dc002870 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f70ec0049b0 con 0x7f70dc002870 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient: dump: 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.065 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 297307563 0 0) 0x7f70dc0065d0 con 0x7f70dc002870 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70dc00a160 con 0x7f70dc002870 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f70ec004c50 con 0x7f70dc002870 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70dc002870 legacy=0x7f70dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.052+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70dc002870 legacy=0x7f70dc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f70dc009e30 con 0x7f70dc0090f0 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70fbfff640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70dc0090f0 legacy=0x7f70dc009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3044677482 0 0) 0x7f70dc009e30 con 0x7f70dc0090f0 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient(hunting): my global_id is 4467 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70dc0065d0 con 0x7f70dc0090f0 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3279494990 0 0) 0x7f70dc0065d0 con 0x7f70dc0090f0 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70dc009e30 con 0x7f70dc0090f0 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f70ec0052f0 con 0x7f70dc0090f0 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient: dump: 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 368984327 0 0) 0x7f70dc009e30 con 0x7f70dc0090f0 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70dc00c3a0 con 0x7f70dc0090f0 2024-04-25T21:40:29.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4131 (unknown 0 0 3400371027) 0x7f70ec0052c0 con 0x7f70dc0090f0 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70dc0090f0 legacy=0x7f70dc009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f70f8ff9640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70dc0090f0 legacy=0x7f70dc009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 16, 2024-04-25T21:40:29.070 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 49231, 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:29.071 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:29.072 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:29.073 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:29.074 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:29.075 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:29.077 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:29.079 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.080 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.081 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:29.081 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.203:6812/3164820986 conn(0x7f70e8036b60 legacy=0x7f70e8039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 10 monclient: shutdown 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 >> v1:172.21.15.6:6789/0 conn(0x7f70fc0a0b60 legacy=0x7f70fc1d4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7100ea4640 1 -- 172.21.15.6:0/213368471 reap_dead start 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7100ea4640 1 -- 172.21.15.6:0/213368471 reap_dead start 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 shutdown_connections 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.060+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 >> 172.21.15.6:0/213368471 conn(0x7f70fc0a1c50 msgr2=0x7f70fc0a4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:29.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.064+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 shutdown_connections 2024-04-25T21:40:29.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.064+0000 7f7102907640 1 -- 172.21.15.6:0/213368471 wait complete. 2024-04-25T21:40:29.083 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-25T21:40:29.083 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-25T21:40:29.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-25T21:40:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.196+0000 7f4b7de80640 10 monclient: get_monmap_and_config 2024-04-25T21:40:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.196+0000 7f4b7de80640 10 monclient: build_initial_monmap 2024-04-25T21:40:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.196+0000 7f4b7de80640 10 monclient: monmap: 2024-04-25T21:40:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:29.201 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:29.203447+0000 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:29.203447+0000 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:29.202 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 1 Processor -- start 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 1 -- start start 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient: init 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient(hunting): picked mon.noname-b con 0x7f4b780a2c20 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient(hunting): picked mon.noname-a con 0x7f4b780a40d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7813d580 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b780a0e50 con 0x7f4b780a40d0 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b780a1130 con 0x7f4b7813d580 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b780a3710 con 0x7f4b780a2c20 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:29.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b7de80640 10 monclient(hunting): authenticate will time out at 988.113468s 2024-04-25T21:40:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b777fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4b780a2c20 0x7f4b780a3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:36014/0 (socket says 172.21.15.6:36014) 2024-04-25T21:40:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b777fe640 1 -- 172.21.15.6:0/2057791694 learned_addr learned my addr 172.21.15.6:0/2057791694 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b76ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b780a40d0 0x7f4b7813be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52466/0 (socket says 172.21.15.6:52466) 2024-04-25T21:40:29.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b77fff640 1 -- 172.21.15.6:0/2057791694 >> v1:172.21.15.203:6789/0 conn(0x7f4b7813d580 legacy=0x7f4b7813f970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b76ffd640 1 -- 172.21.15.6:0/2057791694 >> v1:172.21.15.6:6789/0 conn(0x7f4b780a40d0 legacy=0x7f4b7813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.200+0000 7f4b777fe640 1 -- 172.21.15.6:0/2057791694 >> v1:172.21.15.6:6790/0 conn(0x7f4b780a2c20 legacy=0x7f4b780a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376069102 0 0) 0x7f4b780a0e50 con 0x7f4b780a40d0 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): my global_id is 4470 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b60003530 con 0x7f4b780a40d0 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2397381918 0 0) 0x7f4b780a3710 con 0x7f4b780a2c20 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient(hunting): my global_id is 4439 2024-04-25T21:40:29.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b780a0e50 con 0x7f4b780a2c20 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1619287006 0 0) 0x7f4b60003530 con 0x7f4b780a40d0 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 >> v1:172.21.15.6:6790/0 conn(0x7f4b780a2c20 legacy=0x7f4b780a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 >> v1:172.21.15.203:6789/0 conn(0x7f4b7813d580 legacy=0x7f4b7813f970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 monclient: found mon.noname-a 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7806e9f0 con 0x7f4b780a40d0 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4b680018d0 con 0x7f4b780a40d0 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: dump: 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 5 monclient: authenticate success, global_id 4470 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b68001c20 con 0x7f4b780a40d0 2024-04-25T21:40:29.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 1 -- 172.21.15.6:0/2057791694 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4b68004990 con 0x7f4b780a40d0 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b767fc640 10 monclient: dump: 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 4 set_mon_vals no callback set 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient: shutdown 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- 172.21.15.6:0/2057791694 >> v1:172.21.15.6:6789/0 conn(0x7f4b780a40d0 legacy=0x7f4b7813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- 172.21.15.6:0/2057791694 shutdown_connections 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- 172.21.15.6:0/2057791694 >> 172.21.15.6:0/2057791694 conn(0x7f4b780a5e10 msgr2=0x7f4b780a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- 172.21.15.6:0/2057791694 shutdown_connections 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- 172.21.15.6:0/2057791694 wait complete. 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient: build_initial_monmap 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient: monmap: 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 Processor -- start 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- start start 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient: init 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient(hunting): picked mon.noname-b con 0x7f4b780a2c20 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient(hunting): picked mon.noname-a con 0x7f4b780a40d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:29.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7813d580 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b78140720 con 0x7f4b780a40d0 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b78122260 con 0x7f4b780a2c20 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b78124550 con 0x7f4b7813d580 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b7de80640 10 monclient(hunting): authenticate will time out at 988.119234s 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b76ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b780a40d0 0x7f4b781c8340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52480/0 (socket says 172.21.15.6:52480) 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b76ffd640 1 -- 172.21.15.6:0/3100800128 learned_addr learned my addr 172.21.15.6:0/3100800128 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:29.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.204+0000 7f4b777fe640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.203:6789/0 conn(0x7f4b780a2c20 legacy=0x7f4b7813cbd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b76ffd640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6789/0 conn(0x7f4b780a40d0 legacy=0x7f4b781c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b77fff640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b7813d580 legacy=0x7f4b781d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583151262 0 0) 0x7f4b78140720 con 0x7f4b780a40d0 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): my global_id is 4473 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b64003560 con 0x7f4b780a40d0 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408362798 0 0) 0x7f4b78124550 con 0x7f4b7813d580 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): my global_id is 4442 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b78140720 con 0x7f4b7813d580 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 531860115 0 0) 0x7f4b64003560 con 0x7f4b780a40d0 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b78124550 con 0x7f4b780a40d0 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4b68004990 con 0x7f4b780a40d0 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): dump: 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1237850547 0 0) 0x7f4b78140720 con 0x7f4b7813d580 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b64003560 con 0x7f4b7813d580 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4b70004670 con 0x7f4b7813d580 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient(hunting): dump: 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3901438691 0 0) 0x7f4b78124550 con 0x7f4b780a40d0 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b7813d580 legacy=0x7f4b781d3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.203:6789/0 conn(0x7f4b780a2c20 legacy=0x7f4b7813cbd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 monclient: found mon.a 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7813d2e0 con 0x7f4b780a40d0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: _finish_auth 0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 5 monclient: authenticate success, global_id 4473 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 10 monclient: _renew_subs 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b781c8a50 con 0x7f4b780a40d0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 10 monclient: _renew_subs 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b781d5530 con 0x7f4b780a40d0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b68001c20 con 0x7f4b780a40d0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4b680051a0 con 0x7f4b780a40d0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b75ffb640 4 set_mon_vals no callback set 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 10 monclient: dump: 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f4b6800ff40 con 0x7f4b780a40d0 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4b580065d0 con 0x7f4b58002870 2024-04-25T21:40:29.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3898+0+0 (unknown 2549805797 0 0) 0x7f4b68044610 con 0x7f4b780a40d0 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.208+0000 7f4b77fff640 10 monclient: get_auth_request con 0x7f4b64036aa0 auth_method 0 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b777fe640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b58002870 legacy=0x7f4b58004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2215924339 0 0) 0x7f4b580065d0 con 0x7f4b58002870 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient(hunting): my global_id is 4445 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b78124550 con 0x7f4b58002870 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 976254318 0 0) 0x7f4b78124550 con 0x7f4b58002870 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b580065d0 con 0x7f4b58002870 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4b6c001a60 con 0x7f4b58002870 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient: dump: 2024-04-25T21:40:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 715532181 0 0) 0x7f4b580065d0 con 0x7f4b58002870 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b5800a160 con 0x7f4b58002870 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f4b6c001ab0 con 0x7f4b58002870 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:29.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b58002870 legacy=0x7f4b58004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.212+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b58002870 legacy=0x7f4b58004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b7de80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f4b58009e50 con 0x7f4b58009110 2024-04-25T21:40:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b777fe640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b58009110 legacy=0x7f4b580095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4223131614 0 0) 0x7f4b58009e50 con 0x7f4b58009110 2024-04-25T21:40:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:29.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient(hunting): my global_id is 4448 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b580065d0 con 0x7f4b58009110 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1937647126 0 0) 0x7f4b580065d0 con 0x7f4b58009110 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b58009e50 con 0x7f4b58009110 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4b6c004c10 con 0x7f4b58009110 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient: dump: 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1739501068 0 0) 0x7f4b58009e50 con 0x7f4b58009110 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4b5800c3e0 con 0x7f4b58009110 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4299 (unknown 0 0 2388130025) 0x7f4b6c004e70 con 0x7f4b58009110 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b58009110 legacy=0x7f4b580095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.220+0000 7f4b5ffff640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6790/0 conn(0x7f4b58009110 legacy=0x7f4b580095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:29.230 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 16, 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 49391, 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:29.231 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:29.232 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.233 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:29.234 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.235 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:29.236 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.237 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.238 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.239 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.240 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 5 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:29.241 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:29.242 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.203:6812/3164820986 conn(0x7f4b64036aa0 legacy=0x7f4b64038f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 10 monclient: shutdown 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 >> v1:172.21.15.6:6789/0 conn(0x7f4b780a40d0 legacy=0x7f4b781c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b77fff640 1 -- 172.21.15.6:0/3100800128 reap_dead start 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b77fff640 1 -- 172.21.15.6:0/3100800128 reap_dead start 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 shutdown_connections 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 >> 172.21.15.6:0/3100800128 conn(0x7f4b780a5e10 msgr2=0x7f4b7809aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 shutdown_connections 2024-04-25T21:40:29.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:29.224+0000 7f4b7de80640 1 -- 172.21.15.6:0/3100800128 wait complete. 2024-04-25T21:40:29.244 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-25T21:40:29.244 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-25T21:40:29.884 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[28170]: starting. 2024-04-25T21:40:29.884 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[28170]: creating pool ceph_test_rados_delete_pools_parallel.smithi203-28169 2024-04-25T21:40:29.884 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[28170]: created object 0... 2024-04-25T21:40:29.884 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[28170]: created object 25... 2024-04-25T21:40:29.884 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[28170]: created object 49... 2024-04-25T21:40:29.884 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[28170]: finishing. 2024-04-25T21:40:29.884 INFO:teuthology.orchestra.run.smithi203.stdout:process_1_[28170]: shutting down. 2024-04-25T21:40:31.121 INFO:teuthology.orchestra.run.smithi203.stdout:process_2_[28171]: starting. 2024-04-25T21:40:31.122 INFO:teuthology.orchestra.run.smithi203.stdout:process_2_[28171]: deleting pool ceph_test_rados_delete_pools_parallel.smithi203-28169 2024-04-25T21:40:31.122 INFO:teuthology.orchestra.run.smithi203.stdout:process_2_[28171]: shutting down. 2024-04-25T21:40:33.344 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-04-25T21:40:33.344 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[28204]: starting. 2024-04-25T21:40:33.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[28204]: creating pool ceph_test_rados_delete_pools_parallel.smithi203-28169 2024-04-25T21:40:33.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[28204]: created object 0... 2024-04-25T21:40:33.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[28204]: created object 25... 2024-04-25T21:40:33.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[28204]: created object 49... 2024-04-25T21:40:33.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[28204]: finishing. 2024-04-25T21:40:33.345 INFO:teuthology.orchestra.run.smithi203.stdout:process_3_[28204]: shutting down. 2024-04-25T21:40:33.406 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-04-25T21:40:33.406 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[28206]: starting. 2024-04-25T21:40:33.406 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[28206]: listing objects. 2024-04-25T21:40:33.406 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[28206]: listed object 0... 2024-04-25T21:40:33.406 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[28206]: listed object 25... 2024-04-25T21:40:33.406 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[28206]: saw 50 objects 2024-04-25T21:40:33.406 INFO:teuthology.orchestra.run.smithi203.stdout:process_5_[28206]: shutting down. 2024-04-25T21:40:34.539 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-04-25T21:40:34.539 INFO:teuthology.orchestra.run.smithi203.stdout:process_4_[28205]: starting. 2024-04-25T21:40:34.539 INFO:teuthology.orchestra.run.smithi203.stdout:process_4_[28205]: deleting pool ceph_test_rados_delete_pools_parallel.smithi203-28169 2024-04-25T21:40:34.540 INFO:teuthology.orchestra.run.smithi203.stdout:process_4_[28205]: shutting down. 2024-04-25T21:40:34.542 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-04-25T21:40:34.542 INFO:teuthology.orchestra.run.smithi203.stdout:******************************* 2024-04-25T21:40:34.542 INFO:teuthology.orchestra.run.smithi203.stdout:******* SUCCESS ********** 2024-04-25T21:40:34.545 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-04-25T21:40:34.554 INFO:tasks.mon_thrash:joining mon_thrasher 2024-04-25T21:40:49.245 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-25T21:40:49.245 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-25T21:40:49.245 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T21:40:49.245 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T21:40:49.249 INFO:tasks.ceph.mon.b:Started 2024-04-25T21:40:49.249 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-25T21:40:49.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-25T21:40:49.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.375+0000 7fe572a39640 10 monclient: get_monmap_and_config 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.375+0000 7fe572a39640 10 monclient: build_initial_monmap 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.375+0000 7fe572a39640 10 monclient: monmap: 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:49.382435+0000 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:49.382435+0000 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:49.381 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.375+0000 7fe572a39640 1 Processor -- start 2024-04-25T21:40:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.375+0000 7fe572a39640 1 -- start start 2024-04-25T21:40:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.375+0000 7fe572a39640 10 monclient: init 2024-04-25T21:40:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient(hunting): picked mon.noname-c con 0x7fe56c0a3e80 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient(hunting): picked mon.noname-b con 0x7fe56c0a4970 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient(hunting): picked mon.noname-a con 0x7fe56c09bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56c09c590 con 0x7fe56c09bab0 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56c09adf0 con 0x7fe56c0a3e80 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56c09bf20 con 0x7fe56c0a4970 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient(hunting): authenticate will time out at 1008.292193s 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe56c0a4970 0x7fe56c13c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47296/0 (socket says 172.21.15.6:47296) 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe570fd6640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe56c09bab0 0x7fe56c13f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34570/0 (socket says 172.21.15.6:34570) 2024-04-25T21:40:49.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56b7fe640 1 -- 172.21.15.6:0/125206039 learned_addr learned my addr 172.21.15.6:0/125206039 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe570fd6640 1 -- 172.21.15.6:0/125206039 >> v1:172.21.15.6:6789/0 conn(0x7fe56c09bab0 legacy=0x7fe56c13f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56b7fe640 1 -- 172.21.15.6:0/125206039 >> v1:172.21.15.6:6790/0 conn(0x7fe56c0a4970 legacy=0x7fe56c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56bfff640 1 -- 172.21.15.6:0/125206039 >> v1:172.21.15.203:6789/0 conn(0x7fe56c0a3e80 legacy=0x7fe56c0a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 250395012 0 0) 0x7fe56c09c590 con 0x7fe56c09bab0 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient(hunting): my global_id is 4494 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe554003530 con 0x7fe56c09bab0 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1633535476 0 0) 0x7fe554003530 con 0x7fe56c09bab0 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 >> v1:172.21.15.6:6790/0 conn(0x7fe56c0a4970 legacy=0x7fe56c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 >> v1:172.21.15.203:6789/0 conn(0x7fe56c0a3e80 legacy=0x7fe56c0a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 monclient: found mon.noname-a 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56c06eaf0 con 0x7fe56c09bab0 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: _finish_auth 0 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe564001c50 con 0x7fe56c09bab0 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 5 monclient: authenticate success, global_id 4494 2024-04-25T21:40:49.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: dump: 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe564001fa0 con 0x7fe56c09bab0 2024-04-25T21:40:49.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 1 -- 172.21.15.6:0/125206039 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe564004d10 con 0x7fe56c09bab0 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 4 set_mon_vals no callback set 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe56affd640 10 monclient: dump: 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.390 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.379+0000 7fe572a39640 10 monclient: shutdown 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- 172.21.15.6:0/125206039 >> v1:172.21.15.6:6789/0 conn(0x7fe56c09bab0 legacy=0x7fe56c13f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- 172.21.15.6:0/125206039 shutdown_connections 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- 172.21.15.6:0/125206039 >> 172.21.15.6:0/125206039 conn(0x7fe56c0a5f80 msgr2=0x7fe56c0a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- 172.21.15.6:0/125206039 shutdown_connections 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- 172.21.15.6:0/125206039 wait complete. 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient: build_initial_monmap 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient: monmap: 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:49.391 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 Processor -- start 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- start start 2024-04-25T21:40:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient: init 2024-04-25T21:40:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient(hunting): picked mon.noname-b con 0x7fe56c09bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient(hunting): picked mon.noname-c con 0x7fe56c0a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient(hunting): picked mon.noname-a con 0x7fe56c0a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:49.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56c11fc50 con 0x7fe56c0a4970 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56c13f090 con 0x7fe56c09bab0 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56c120bf0 con 0x7fe56c0a3e80 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe572a39640 10 monclient(hunting): authenticate will time out at 1008.297582s 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe56b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe56c0a3e80 0x7fe56c13dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47308/0 (socket says 172.21.15.6:47308) 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe56b7fe640 1 -- 172.21.15.6:0/1323901568 learned_addr learned my addr 172.21.15.6:0/1323901568 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe56b7fe640 1 -- 172.21.15.6:0/1323901568 >> v1:172.21.15.6:6790/0 conn(0x7fe56c0a3e80 legacy=0x7fe56c13dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.383+0000 7fe570fd6640 1 -- 172.21.15.6:0/1323901568 >> v1:172.21.15.6:6789/0 conn(0x7fe56c0a4970 legacy=0x7fe56c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 366318535 0 0) 0x7fe56c11fc50 con 0x7fe56c0a4970 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): my global_id is 4497 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe56bfff640 1 -- 172.21.15.6:0/1323901568 >> v1:172.21.15.203:6789/0 conn(0x7fe56c09bab0 legacy=0x7fe56c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe558003530 con 0x7fe56c0a4970 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1681511863 0 0) 0x7fe558003530 con 0x7fe56c0a4970 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe56c11fc50 con 0x7fe56c0a4970 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe564004d10 con 0x7fe56c0a4970 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.397 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2466076057 0 0) 0x7fe56c11fc50 con 0x7fe56c0a4970 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 >> v1:172.21.15.6:6790/0 conn(0x7fe56c0a3e80 legacy=0x7fe56c13dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 >> v1:172.21.15.203:6789/0 conn(0x7fe56c09bab0 legacy=0x7fe56c13d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 monclient: found mon.a 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56c120630 con 0x7fe56c0a4970 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:49.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 5 monclient: authenticate success, global_id 4497 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 10 monclient: _renew_subs 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe56c120250 con 0x7fe56c0a4970 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 10 monclient: _renew_subs 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe56c1d5000 con 0x7fe56c0a4970 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe564001fa0 con 0x7fe56c0a4970 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe564005520 con 0x7fe56c0a4970 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 10 monclient: dump: 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.400 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe56a7fc640 4 set_mon_vals no callback set 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fe5640102c0 con 0x7fe56c0a4970 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe54c005090 con 0x7fe56c0a4970 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fe564044680 con 0x7fe56c0a4970 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.387+0000 7fe56bfff640 10 monclient: get_auth_request con 0x7fe558036b00 auth_method 0 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.391+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fe564015a60 con 0x7fe56c0a4970 2024-04-25T21:40:49.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.391+0000 7fe568ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:49.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.391+0000 7fe568ff9640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:49.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.611+0000 7fe572a39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-25T21:40:49.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.611+0000 7fe572a39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:49.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.611+0000 7fe572a39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.611+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe54c005380 con 0x7fe56c0a4970 2024-04-25T21:40:49.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe568ff9640 1 -- 172.21.15.6:0/1323901568 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 391888619) 0x7fe56401df10 con 0x7fe56c0a4970 2024-04-25T21:40:49.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe568ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:49.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe568ff9640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:49.618 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","modified":"2024-04-25T21:39:14.743126Z","created":"2024-04-25T21:39:14.743126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-25T21:40:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 >> v1:172.21.15.203:6812/3164820986 conn(0x7fe558036b00 legacy=0x7fe558038fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 10 monclient: shutdown 2024-04-25T21:40:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 >> v1:172.21.15.6:6789/0 conn(0x7fe56c0a4970 legacy=0x7fe56c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 shutdown_connections 2024-04-25T21:40:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 >> 172.21.15.6:0/1323901568 conn(0x7fe56c0a5f80 msgr2=0x7fe56c140380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 shutdown_connections 2024-04-25T21:40:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.615+0000 7fe572a39640 1 -- 172.21.15.6:0/1323901568 wait complete. 2024-04-25T21:40:49.635 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-25T21:40:49.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-25T21:40:49.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient: get_monmap_and_config 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient: build_initial_monmap 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient: monmap: 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:49.760515+0000 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:49.760515+0000 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:49.759 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 1 Processor -- start 2024-04-25T21:40:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 1 -- start start 2024-04-25T21:40:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient: init 2024-04-25T21:40:49.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient(hunting): picked mon.noname-c con 0x7feac409e860 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient(hunting): picked mon.noname-b con 0x7feac40a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient(hunting): picked mon.noname-a con 0x7feac40a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac409c130 con 0x7feac40a1680 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac4099520 con 0x7feac409e860 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac4099b10 con 0x7feac40a1270 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac96f2640 10 monclient(hunting): authenticate will time out at 1008.670349s 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac27fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feac40a1270 0x7feac4139b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47314/0 (socket says 172.21.15.6:47314) 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.755+0000 7feac27fc640 1 -- 172.21.15.6:0/2875067460 learned_addr learned my addr 172.21.15.6:0/2875067460 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac37fe640 1 -- 172.21.15.6:0/2875067460 >> v1:172.21.15.6:6789/0 conn(0x7feac40a1680 legacy=0x7feac413d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac27fc640 1 -- 172.21.15.6:0/2875067460 >> v1:172.21.15.6:6790/0 conn(0x7feac40a1270 legacy=0x7feac4139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2040477236 0 0) 0x7feac4099b10 con 0x7feac40a1270 2024-04-25T21:40:49.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): my global_id is 4460 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaac003530 con 0x7feac40a1270 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586857521 0 0) 0x7feabc001830 con 0x7feac40a1680 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient(hunting): my global_id is 4500 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac4099b10 con 0x7feac40a1680 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac2ffd640 1 -- 172.21.15.6:0/2875067460 >> v1:172.21.15.203:6789/0 conn(0x7feac409e860 legacy=0x7feac40a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 702420170 0 0) 0x7feac4099b10 con 0x7feac40a1680 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 >> v1:172.21.15.6:6790/0 conn(0x7feac40a1270 legacy=0x7feac4139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 >> v1:172.21.15.203:6789/0 conn(0x7feac409e860 legacy=0x7feac40a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 monclient: found mon.noname-a 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac409a0d0 con 0x7feac40a1680 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7feabc001eb0 con 0x7feac40a1680 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: dump: 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 5 monclient: authenticate success, global_id 4500 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feabc0039a0 con 0x7feac40a1680 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 1 -- 172.21.15.6:0/2875067460 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7feabc003cf0 con 0x7feac40a1680 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac1ffb640 10 monclient: dump: 2024-04-25T21:40:49.770 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 4 set_mon_vals no callback set 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient: shutdown 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- 172.21.15.6:0/2875067460 >> v1:172.21.15.6:6789/0 conn(0x7feac40a1680 legacy=0x7feac413d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- 172.21.15.6:0/2875067460 shutdown_connections 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- 172.21.15.6:0/2875067460 >> 172.21.15.6:0/2875067460 conn(0x7feac40a1c50 msgr2=0x7feac40a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- 172.21.15.6:0/2875067460 shutdown_connections 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- 172.21.15.6:0/2875067460 wait complete. 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient: build_initial_monmap 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient: monmap: 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:49.772 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 Processor -- start 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- start start 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient: init 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient(hunting): picked mon.noname-b con 0x7feac409e860 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient(hunting): picked mon.noname-c con 0x7feac40a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient(hunting): picked mon.noname-a con 0x7feac40a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac4093af0 con 0x7feac40a1680 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac4072b40 con 0x7feac409e860 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac40920e0 con 0x7feac40a1270 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:49.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac96f2640 10 monclient(hunting): authenticate will time out at 1008.675850s 2024-04-25T21:40:49.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac27fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7feac40a1270 0x7feac40933e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47326/0 (socket says 172.21.15.6:47326) 2024-04-25T21:40:49.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac37fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feac40a1680 0x7feac41d1ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34612/0 (socket says 172.21.15.6:34612) 2024-04-25T21:40:49.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.759+0000 7feac27fc640 1 -- 172.21.15.6:0/384270378 learned_addr learned my addr 172.21.15.6:0/384270378 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac27fc640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6790/0 conn(0x7feac40a1270 legacy=0x7feac40933e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac37fe640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feac40a1680 legacy=0x7feac41d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac2ffd640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.203:6789/0 conn(0x7feac409e860 legacy=0x7feac4092cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2059882762 0 0) 0x7feac40920e0 con 0x7feac40a1270 2024-04-25T21:40:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): my global_id is 4463 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab0003530 con 0x7feac40a1270 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3222341082 0 0) 0x7feac4093af0 con 0x7feac40a1680 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): my global_id is 4503 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac40920e0 con 0x7feac40a1680 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1834923822 0 0) 0x7feab0003530 con 0x7feac40a1270 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feac4093af0 con 0x7feac40a1270 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7feab80047f0 con 0x7feac40a1270 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): dump: 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3180867504 0 0) 0x7feac40920e0 con 0x7feac40a1680 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feab0003530 con 0x7feac40a1680 2024-04-25T21:40:49.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7feabc003cf0 con 0x7feac40a1680 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient(hunting): dump: 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3331754121 0 0) 0x7feac4093af0 con 0x7feac40a1270 2024-04-25T21:40:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.203:6789/0 conn(0x7feac409e860 legacy=0x7feac4092cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feac40a1680 legacy=0x7feac41d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 monclient: found mon.c 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac4094880 con 0x7feac40a1270 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 5 monclient: authenticate success, global_id 4463 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 10 monclient: _renew_subs 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feac4096340 con 0x7feac40a1270 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 10 monclient: _renew_subs 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feac4097360 con 0x7feac40a1270 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feab8004c40 con 0x7feac40a1270 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7feab8004f90 con 0x7feac40a1270 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac17fa640 4 set_mon_vals no callback set 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 10 monclient: dump: 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.782 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feaa40065d0 con 0x7feaa4002870 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.763+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7feab8005170 con 0x7feac40a1270 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7feab8044530 con 0x7feac40a1270 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feac37fe640 10 monclient: get_auth_request con 0x7feab0036a70 auth_method 0 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feac2ffd640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feaa4002870 legacy=0x7feaa4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2811212037 0 0) 0x7feaa40065d0 con 0x7feaa4002870 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient(hunting): my global_id is 4506 2024-04-25T21:40:49.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac4093af0 con 0x7feaa4002870 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3767202174 0 0) 0x7feac4093af0 con 0x7feaa4002870 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feaa40065d0 con 0x7feaa4002870 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7feab4001990 con 0x7feaa4002870 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient: dump: 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.784 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 358517510 0 0) 0x7feaa40065d0 con 0x7feaa4002870 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feaa400a160 con 0x7feaa4002870 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7feab4001880 con 0x7feaa4002870 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feaa4002870 legacy=0x7feaa4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.767+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feaa4002870 legacy=0x7feaa4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:49.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feac96f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7feaa4009e50 con 0x7feaa4009110 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feac2ffd640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feaa4009110 legacy=0x7feaa40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953252710 0 0) 0x7feaa4009e50 con 0x7feaa4009110 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 10 monclient(hunting): my global_id is 4509 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaa40065d0 con 0x7feaa4009110 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1771869064 0 0) 0x7feaa40065d0 con 0x7feaa4009110 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feaa4009e50 con 0x7feaa4009110 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7feab40039b0 con 0x7feaa4009110 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 10 monclient: dump: 2024-04-25T21:40:49.787 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 914198976 0 0) 0x7feaa4009e50 con 0x7feaa4009110 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.775+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 --> v1:172.21.15.6:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feaa400c3e0 con 0x7feaa4009110 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 <== mon.0 v1:172.21.15.6:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4142 (unknown 0 0 2185330734) 0x7feab40039b0 con 0x7feaa4009110 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feaab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feaab7fe640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feaa4009110 legacy=0x7feaa40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feaab7fe640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6789/0 conn(0x7feaa4009110 legacy=0x7feaa40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "leader", 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 18, 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 69950, 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:49.789 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:49.790 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:49.791 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:49.792 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:49.793 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:49.794 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.795 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:49.796 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:49.797 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:49.798 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 2 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:49.799 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:49.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.203:6812/3164820986 conn(0x7feab0036a70 legacy=0x7feab0038f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 10 monclient: shutdown 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 >> v1:172.21.15.6:6790/0 conn(0x7feac40a1270 legacy=0x7feac40933e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac37fe640 1 -- 172.21.15.6:0/384270378 reap_dead start 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac37fe640 1 -- 172.21.15.6:0/384270378 reap_dead start 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 shutdown_connections 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 >> 172.21.15.6:0/384270378 conn(0x7feac40a1c50 msgr2=0x7feac40987f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 shutdown_connections 2024-04-25T21:40:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.779+0000 7feac96f2640 1 -- 172.21.15.6:0/384270378 wait complete. 2024-04-25T21:40:49.801 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-25T21:40:49.801 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-25T21:40:49.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-25T21:40:49.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient: get_monmap_and_config 2024-04-25T21:40:49.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient: build_initial_monmap 2024-04-25T21:40:49.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient: monmap: 2024-04-25T21:40:49.918 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:49.918 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.918 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:49.919747+0000 2024-04-25T21:40:49.918 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:49.919747+0000 2024-04-25T21:40:49.919 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:49.919 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.919 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:49.919 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:49.919 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:49.919 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 1 Processor -- start 2024-04-25T21:40:49.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 1 -- start start 2024-04-25T21:40:49.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient: init 2024-04-25T21:40:49.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:49.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:49.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient(hunting): picked mon.noname-c con 0x7f348006e890 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:49.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient(hunting): picked mon.noname-a con 0x7f348009bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient(hunting): picked mon.noname-b con 0x7f348013d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f348009b0d0 con 0x7f348009bab0 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34800a4ca0 con 0x7f348006e890 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f348006f070 con 0x7f348013d0a0 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f3486dc2640 10 monclient(hunting): authenticate will time out at 1008.829560s 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f348535f640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f348013d0a0 0x7f348013f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47332/0 (socket says 172.21.15.6:47332) 2024-04-25T21:40:49.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.915+0000 7f348535f640 1 -- 172.21.15.6:0/306455571 learned_addr learned my addr 172.21.15.6:0/306455571 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f348535f640 1 -- 172.21.15.6:0/306455571 >> v1:172.21.15.6:6790/0 conn(0x7f348013d0a0 legacy=0x7f348013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347ffff640 1 -- 172.21.15.6:0/306455571 >> v1:172.21.15.6:6789/0 conn(0x7f348009bab0 legacy=0x7f34800a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3484b5e640 1 -- 172.21.15.6:0/306455571 >> v1:172.21.15.203:6789/0 conn(0x7f348006e890 legacy=0x7f34800a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3688038784 0 0) 0x7f348006f070 con 0x7f348013d0a0 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): my global_id is 4466 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f346c003530 con 0x7f348013d0a0 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965093022 0 0) 0x7f34680014b0 con 0x7f348009bab0 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): my global_id is 4512 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f348006f070 con 0x7f348009bab0 2024-04-25T21:40:49.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1201374401 0 0) 0x7f34800a4ca0 con 0x7f348006e890 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient(hunting): my global_id is 94102 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34680014b0 con 0x7f348006e890 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1062898936 0 0) 0x7f348006f070 con 0x7f348009bab0 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 >> v1:172.21.15.6:6790/0 conn(0x7f348013d0a0 legacy=0x7f348013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 >> v1:172.21.15.203:6789/0 conn(0x7f348006e890 legacy=0x7f34800a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 monclient: found mon.noname-a 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348009adf0 con 0x7f348009bab0 2024-04-25T21:40:49.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3468001af0 con 0x7f348009bab0 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: dump: 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.926 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 5 monclient: authenticate success, global_id 4512 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3468003620 con 0x7f348009bab0 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 4 set_mon_vals no callback set 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 1 -- 172.21.15.6:0/306455571 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f34680039b0 con 0x7f348009bab0 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347f7fe640 10 monclient: dump: 2024-04-25T21:40:49.927 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient: shutdown 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- 172.21.15.6:0/306455571 >> v1:172.21.15.6:6789/0 conn(0x7f348009bab0 legacy=0x7f34800a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- 172.21.15.6:0/306455571 shutdown_connections 2024-04-25T21:40:49.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- 172.21.15.6:0/306455571 >> 172.21.15.6:0/306455571 conn(0x7f34800a5f80 msgr2=0x7f34800a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:49.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- 172.21.15.6:0/306455571 shutdown_connections 2024-04-25T21:40:49.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- 172.21.15.6:0/306455571 wait complete. 2024-04-25T21:40:49.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient: build_initial_monmap 2024-04-25T21:40:49.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient: monmap: 2024-04-25T21:40:49.933 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 Processor -- start 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- start start 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient: init 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:49.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient(hunting): picked mon.noname-b con 0x7f348006e890 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient(hunting): picked mon.noname-a con 0x7f348009bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient(hunting): picked mon.noname-c con 0x7f348013d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34800a2530 con 0x7f348009bab0 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f348009c490 con 0x7f348006e890 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34800a9070 con 0x7f348013d0a0 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f3486dc2640 10 monclient(hunting): authenticate will time out at 1008.835355s 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f348009bab0 0x7f348013c670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34638/0 (socket says 172.21.15.6:34638) 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.919+0000 7f347ffff640 1 -- 172.21.15.6:0/54255743 learned_addr learned my addr 172.21.15.6:0/54255743 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347ffff640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6789/0 conn(0x7f348009bab0 legacy=0x7f348013c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f348535f640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f348013d0a0 legacy=0x7f34801d3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553687965 0 0) 0x7f34800a2530 con 0x7f348009bab0 2024-04-25T21:40:49.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): my global_id is 4515 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3460003530 con 0x7f348009bab0 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202043034 0 0) 0x7f34800a9070 con 0x7f348013d0a0 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): my global_id is 4469 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3484b5e640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.203:6789/0 conn(0x7f348006e890 legacy=0x7f34801cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34800a2530 con 0x7f348013d0a0 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 547468012 0 0) 0x7f3460003530 con 0x7f348009bab0 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34800a9070 con 0x7f348009bab0 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3468001ab0 con 0x7f348009bab0 2024-04-25T21:40:49.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2400940467 0 0) 0x7f34800a2530 con 0x7f348013d0a0 2024-04-25T21:40:49.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3460003530 con 0x7f348013d0a0 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269288632 0 0) 0x7f348009c490 con 0x7f348006e890 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): my global_id is 94105 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34800a2530 con 0x7f348006e890 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3478001fc0 con 0x7f348013d0a0 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.938 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3343883679 0 0) 0x7f34800a9070 con 0x7f348009bab0 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f348013d0a0 legacy=0x7f34801d3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.203:6789/0 conn(0x7f348006e890 legacy=0x7f34801cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 monclient: found mon.a 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348013cd80 con 0x7f348009bab0 2024-04-25T21:40:49.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 5 monclient: authenticate success, global_id 4515 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 10 monclient: _renew_subs 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f34801373a0 con 0x7f348009bab0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 10 monclient: _renew_subs 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f34801d4e90 con 0x7f348009bab0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3468003f70 con 0x7f348009bab0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f34680042c0 con 0x7f348009bab0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347effd640 4 set_mon_vals no callback set 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 10 monclient: dump: 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f34640065d0 con 0x7f3464002870 2024-04-25T21:40:49.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f34680103d0 con 0x7f348009bab0 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f3468044660 con 0x7f348009bab0 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.923+0000 7f348535f640 10 monclient: get_auth_request con 0x7f3460036c40 auth_method 0 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f3484b5e640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f3464002870 legacy=0x7f3464004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450335069 0 0) 0x7f34640065d0 con 0x7f3464002870 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient(hunting): my global_id is 4472 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34800a9070 con 0x7f3464002870 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 441846765 0 0) 0x7f34800a9070 con 0x7f3464002870 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f34640065d0 con 0x7f3464002870 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3470001a50 con 0x7f3464002870 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:49.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient: dump: 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3983240989 0 0) 0x7f34640065d0 con 0x7f3464002870 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f346400a160 con 0x7f3464002870 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f34700057d0 con 0x7f3464002870 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:49.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f3464002870 legacy=0x7f3464004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.927+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f3464002870 legacy=0x7f3464004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f3486dc2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3464009e30 con 0x7f34640090f0 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f3484b5e640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f34640090f0 legacy=0x7f3464009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 272642350 0 0) 0x7f3464009e30 con 0x7f34640090f0 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient(hunting): my global_id is 4475 2024-04-25T21:40:49.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34640065d0 con 0x7f34640090f0 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2653745592 0 0) 0x7f34640065d0 con 0x7f34640090f0 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3464009e30 con 0x7f34640090f0 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f34700037c0 con 0x7f34640090f0 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient: dump: 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:49.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 167669567 0 0) 0x7f3464009e30 con 0x7f34640090f0 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 --> v1:172.21.15.6:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f346400c3a0 con 0x7f34640090f0 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 <== mon.2 v1:172.21.15.6:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4310 (unknown 0 0 3442892977) 0x7f3470005980 con 0x7f34640090f0 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f34640090f0 legacy=0x7f3464009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.935+0000 7f347d7fa640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6790/0 conn(0x7f34640090f0 legacy=0x7f3464009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:49.947 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 18, 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 70106, 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:49.948 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:49.949 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:49.950 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:49.951 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:49.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:49.954 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.955 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.956 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": [ 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 3 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.957 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 4 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": [ 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:49.958 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:49.959 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.203:6812/3164820986 conn(0x7f3460036c40 legacy=0x7f3460039100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 10 monclient: shutdown 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 >> v1:172.21.15.6:6789/0 conn(0x7f348009bab0 legacy=0x7f348013c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f348535f640 1 -- 172.21.15.6:0/54255743 reap_dead start 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f348535f640 1 -- 172.21.15.6:0/54255743 reap_dead start 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 shutdown_connections 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 >> 172.21.15.6:0/54255743 conn(0x7f34800a5f80 msgr2=0x7f3480136060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 shutdown_connections 2024-04-25T21:40:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:49.939+0000 7f3486dc2640 1 -- 172.21.15.6:0/54255743 wait complete. 2024-04-25T21:40:49.961 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-25T21:40:49.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-25T21:40:49.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-25T21:40:50.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient: get_monmap_and_config 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient: build_initial_monmap 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient: monmap: 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:50.083755+0000 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:50.083755+0000 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 1 Processor -- start 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 1 -- start start 2024-04-25T21:40:50.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient: init 2024-04-25T21:40:50.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:50.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:50.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient(hunting): picked mon.noname-a con 0x7f237006e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:50.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient(hunting): picked mon.noname-c con 0x7f237009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient(hunting): picked mon.noname-b con 0x7f237013d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f237009b0d0 con 0x7f237006e890 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23700a4ca0 con 0x7f237009bab0 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f237006f070 con 0x7f237013d0a0 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f2376c42640 10 monclient(hunting): authenticate will time out at 1008.993628s 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f23749de640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f237006e890 0x7f23700a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34654/0 (socket says 172.21.15.6:34654) 2024-04-25T21:40:50.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f23749de640 1 -- 172.21.15.6:0/2958280533 learned_addr learned my addr 172.21.15.6:0/2958280533 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:50.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f23749de640 1 -- 172.21.15.6:0/2958280533 >> v1:172.21.15.6:6789/0 conn(0x7f237006e890 legacy=0x7f23700a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.079+0000 7f23751df640 1 -- 172.21.15.6:0/2958280533 >> v1:172.21.15.6:6790/0 conn(0x7f237013d0a0 legacy=0x7f237013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394058749 0 0) 0x7f2358001460 con 0x7f237006e890 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): my global_id is 4518 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236ffff640 1 -- 172.21.15.6:0/2958280533 >> v1:172.21.15.203:6789/0 conn(0x7f237009bab0 legacy=0x7f23700a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f235c003530 con 0x7f237006e890 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2758627030 0 0) 0x7f237006f070 con 0x7f237013d0a0 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient(hunting): my global_id is 4478 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2358001460 con 0x7f237013d0a0 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 391055614 0 0) 0x7f2358001460 con 0x7f237013d0a0 2024-04-25T21:40:50.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 >> v1:172.21.15.203:6789/0 conn(0x7f237009bab0 legacy=0x7f23700a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 >> v1:172.21.15.6:6789/0 conn(0x7f237006e890 legacy=0x7f23700a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 monclient: found mon.noname-b 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237009adf0 con 0x7f237013d0a0 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2368001ca0 con 0x7f237013d0a0 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: dump: 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.089 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 5 monclient: authenticate success, global_id 4478 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2368001ff0 con 0x7f237013d0a0 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 1 -- 172.21.15.6:0/2958280533 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2368003b40 con 0x7f237013d0a0 2024-04-25T21:40:50.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 4 set_mon_vals no callback set 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236f7fe640 10 monclient: dump: 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient: shutdown 2024-04-25T21:40:50.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- 172.21.15.6:0/2958280533 >> v1:172.21.15.6:6790/0 conn(0x7f237013d0a0 legacy=0x7f237013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- 172.21.15.6:0/2958280533 shutdown_connections 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- 172.21.15.6:0/2958280533 >> 172.21.15.6:0/2958280533 conn(0x7f23700a5f80 msgr2=0x7f23700a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- 172.21.15.6:0/2958280533 shutdown_connections 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- 172.21.15.6:0/2958280533 wait complete. 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient: build_initial_monmap 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient: monmap: 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:50.092 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:50.093 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:50.093 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 Processor -- start 2024-04-25T21:40:50.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- start start 2024-04-25T21:40:50.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient: init 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient(hunting): picked mon.noname-b con 0x7f237006e890 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient(hunting): picked mon.noname-a con 0x7f237009bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient(hunting): picked mon.noname-c con 0x7f237013d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23700a1ff0 con 0x7f237009bab0 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23700a3860 con 0x7f237006e890 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23700a9090 con 0x7f237013d0a0 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f2376c42640 10 monclient(hunting): authenticate will time out at 1008.998647s 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236ffff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f237009bab0 0x7f23701d07a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34670/0 (socket says 172.21.15.6:34670) 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.083+0000 7f236ffff640 1 -- 172.21.15.6:0/2177427428 learned_addr learned my addr 172.21.15.6:0/2177427428 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:50.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236ffff640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.6:6789/0 conn(0x7f237009bab0 legacy=0x7f23701d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f23751df640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.6:6790/0 conn(0x7f237013d0a0 legacy=0x7f23701d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869546232 0 0) 0x7f23700a1ff0 con 0x7f237009bab0 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): my global_id is 4521 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2350003530 con 0x7f237009bab0 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2909320948 0 0) 0x7f23700a9090 con 0x7f237013d0a0 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): my global_id is 4481 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23700a1ff0 con 0x7f237013d0a0 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f23749de640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f237006e890 legacy=0x7f237013c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2654065552 0 0) 0x7f2350003530 con 0x7f237009bab0 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23700a9090 con 0x7f237009bab0 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2360001c10 con 0x7f237009bab0 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.100 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2719766530 0 0) 0x7f23700a1ff0 con 0x7f237013d0a0 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2350003530 con 0x7f237013d0a0 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2368003b40 con 0x7f237013d0a0 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1964383096 0 0) 0x7f23700a3860 con 0x7f237006e890 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient(hunting): my global_id is 94111 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23700a1ff0 con 0x7f237006e890 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2927650987 0 0) 0x7f23700a9090 con 0x7f237009bab0 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.6:6790/0 conn(0x7f237013d0a0 legacy=0x7f23701d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f237006e890 legacy=0x7f237013c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 monclient: found mon.a 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:50.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2370137950 con 0x7f237009bab0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 5 monclient: authenticate success, global_id 4521 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 10 monclient: _renew_subs 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2370137c30 con 0x7f237009bab0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 10 monclient: _renew_subs 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2370137310 con 0x7f237009bab0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2360003a50 con 0x7f237009bab0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2360003da0 con 0x7f237009bab0 2024-04-25T21:40:50.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236effd640 4 set_mon_vals no callback set 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 10 monclient: dump: 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:50.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f23540065d0 con 0x7f2354002870 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f2360003f40 con 0x7f237009bab0 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f2360044420 con 0x7f237009bab0 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.087+0000 7f23751df640 10 monclient: get_auth_request con 0x7f2350036ce0 auth_method 0 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f23749de640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f2354002870 legacy=0x7f2354004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95780075 0 0) 0x7f23540065d0 con 0x7f2354002870 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient(hunting): my global_id is 94114 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23700a9090 con 0x7f2354002870 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2790626072 0 0) 0x7f23700a9090 con 0x7f2354002870 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f23540065d0 con 0x7f2354002870 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f23580018c0 con 0x7f2354002870 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient: dump: 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2972515474 0 0) 0x7f23540065d0 con 0x7f2354002870 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f235400a160 con 0x7f2354002870 2024-04-25T21:40:50.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f23580017b0 con 0x7f2354002870 2024-04-25T21:40:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f2354002870 legacy=0x7f2354004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.091+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f2354002870 legacy=0x7f2354004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:50.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.099+0000 7f2376c42640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-25T21:40:50.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.099+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2354009e50 con 0x7f2354009110 2024-04-25T21:40:50.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.099+0000 7f23749de640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f2354009110 legacy=0x7f23540095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288686947 0 0) 0x7f2354009e50 con 0x7f2354009110 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient(hunting): my global_id is 94117 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23540065d0 con 0x7f2354009110 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 642613508 0 0) 0x7f23540065d0 con 0x7f2354009110 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2354009e50 con 0x7f2354009110 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2358004c10 con 0x7f2354009110 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient: dump: 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.109 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 312231939 0 0) 0x7f2354009e50 con 0x7f2354009110 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f235400c3e0 con 0x7f2354009110 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 2510194930) 0x7f2358005f90 con 0x7f2354009110 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f2354009110 legacy=0x7f23540095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f236d7fa640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6789/0 conn(0x7f2354009110 legacy=0x7f23540095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "peon", 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: "election_epoch": 18, 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 771, 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum": [ 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: 1, 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: 2 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:50.111 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_age": 0, 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "required_con": "2450521708007866372", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "required_mon": [ 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:50.112 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_con": "4540701547738038271", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "quorum_mon": [ 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:50.113 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "outside_quorum": [], 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "extra_probe_peers": [], 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "sync_provider": [], 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": { 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 1, 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "7c01789f-5916-4cc5-8455-2e5d8f8fad2b", 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-04-25T21:39:14.743126Z", 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release": 19, 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "min_mon_release_name": "squid", 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "election_strategy": 3, 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "disallowed_leaders: ": "", 2024-04-25T21:40:50.114 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false, 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "tiebreaker_mon": "", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_ranks: ": "", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "features": { 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "persistent": [ 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "kraken", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "luminous", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "mimic", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap-prune", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "nautilus", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "octopus", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "pacific", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "elector-pinging", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "quincy", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "reef", 2024-04-25T21:40:50.115 INFO:teuthology.orchestra.run.smithi006.stdout: "squid" 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "optional": [] 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mons": [ 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 0, 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "a", 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789", 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:50.116 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6789/0", 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6789/0", 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 1, 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "b", 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:50.117 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789", 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.203:6789/0", 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.203:6789/0", 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "rank": 2, 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "c", 2024-04-25T21:40:50.118 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790", 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 0 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6790/0", 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6790/0", 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "priority": 0, 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "{}" 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:50.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "feature_map": { 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": [ 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "client": [ 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "features": "0x3f03cffffffdffff", 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "release": "luminous", 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "num": 1 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-04-25T21:40:50.120 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": false 2024-04-25T21:40:50.121 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:50.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.203:6812/3164820986 conn(0x7f2350036ce0 legacy=0x7f23500391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 10 monclient: shutdown 2024-04-25T21:40:50.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:50.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 >> v1:172.21.15.6:6789/0 conn(0x7f237009bab0 legacy=0x7f23701d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f23751df640 1 -- 172.21.15.6:0/2177427428 reap_dead start 2024-04-25T21:40:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f23751df640 1 -- 172.21.15.6:0/2177427428 reap_dead start 2024-04-25T21:40:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 shutdown_connections 2024-04-25T21:40:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 >> 172.21.15.6:0/2177427428 conn(0x7f23700a5f80 msgr2=0x7f2370135fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 shutdown_connections 2024-04-25T21:40:50.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.103+0000 7f2376c42640 1 -- 172.21.15.6:0/2177427428 wait complete. 2024-04-25T21:40:50.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-25T21:40:50.123 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-25T21:40:50.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.223+0000 7f42f0dc4640 10 monclient: get_monmap_and_config 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.223+0000 7f42f0dc4640 10 monclient: build_initial_monmap 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.223+0000 7f42f0dc4640 10 monclient: monmap: 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:50.229776+0000 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:50.229776+0000 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:50.228 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.229 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:50.229 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:50.229 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:50.229 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.223+0000 7f42f0dc4640 1 Processor -- start 2024-04-25T21:40:50.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.223+0000 7f42f0dc4640 1 -- start start 2024-04-25T21:40:50.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.223+0000 7f42f0dc4640 10 monclient: init 2024-04-25T21:40:50.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:50.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f42ec09d8e0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f42ec09fe90 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f42ec09e3d0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ec09f520 con 0x7f42ec09fe90 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ec0a3d70 con 0x7f42ec09d8e0 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ec09ec10 con 0x7f42ec09e3d0 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient(hunting): authenticate will time out at 1009.140338s 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42ead9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f42ec09e3d0 0x7f42ec13c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47380/0 (socket says 172.21.15.6:47380) 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42ead9d640 1 -- 172.21.15.6:0/748563520 learned_addr learned my addr 172.21.15.6:0/748563520 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:50.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e9d9b640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42ec09fe90 0x7f42ec138d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34676/0 (socket says 172.21.15.6:34676) 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42ead9d640 1 -- 172.21.15.6:0/748563520 >> v1:172.21.15.6:6790/0 conn(0x7f42ec09e3d0 legacy=0x7f42ec13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e9d9b640 1 -- 172.21.15.6:0/748563520 >> v1:172.21.15.6:6789/0 conn(0x7f42ec09fe90 legacy=0x7f42ec138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 782218198 0 0) 0x7f42ec09ec10 con 0x7f42ec09e3d0 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): my global_id is 4484 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42d4003530 con 0x7f42ec09e3d0 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2946267082 0 0) 0x7f42ec09f520 con 0x7f42ec09fe90 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient(hunting): my global_id is 4524 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ec09ec10 con 0x7f42ec09fe90 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42ea59c640 1 -- 172.21.15.6:0/748563520 >> v1:172.21.15.203:6789/0 conn(0x7f42ec09d8e0 legacy=0x7f42ec09dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 926488879 0 0) 0x7f42d4003530 con 0x7f42ec09e3d0 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 >> v1:172.21.15.203:6789/0 conn(0x7f42ec09d8e0 legacy=0x7f42ec09dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 >> v1:172.21.15.6:6789/0 conn(0x7f42ec09fe90 legacy=0x7f42ec138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: _finish_hunting 0 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 monclient: found mon.noname-b 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ec0a2050 con 0x7f42ec09e3d0 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: _finish_auth 0 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: _check_auth_tickets 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f42e4001d10 con 0x7f42ec09e3d0 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 5 monclient: authenticate success, global_id 4484 2024-04-25T21:40:50.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: dump: 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:50.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42e4002030 con 0x7f42ec09e3d0 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 1 -- 172.21.15.6:0/748563520 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f42e4004dd0 con 0x7f42ec09e3d0 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 4 set_mon_vals no callback set 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42e959a640 10 monclient: dump: 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.238 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 10 monclient: shutdown 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 1 -- 172.21.15.6:0/748563520 >> v1:172.21.15.6:6790/0 conn(0x7f42ec09e3d0 legacy=0x7f42ec13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 1 -- 172.21.15.6:0/748563520 shutdown_connections 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.227+0000 7f42f0dc4640 1 -- 172.21.15.6:0/748563520 >> 172.21.15.6:0/748563520 conn(0x7f42ec0a2ee0 msgr2=0x7f42ec0a32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 1 -- 172.21.15.6:0/748563520 shutdown_connections 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 1 -- 172.21.15.6:0/748563520 wait complete. 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient: build_initial_monmap 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient: monmap: 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:50.239 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 1 Processor -- start 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 1 -- start start 2024-04-25T21:40:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient: init 2024-04-25T21:40:50.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:50.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:50.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f42ec09d8e0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:50.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f42ec09e3d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:50.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f42ec09fe90 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ec098780 con 0x7f42ec09d8e0 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ec08cea0 con 0x7f42ec09e3d0 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ec08d180 con 0x7f42ec09fe90 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42f0dc4640 10 monclient(hunting): authenticate will time out at 1009.146128s 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42ea59c640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42ec09d8e0 0x7f42ec097960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34684/0 (socket says 172.21.15.6:34684) 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42ea59c640 1 -- 172.21.15.6:0/1100359056 learned_addr learned my addr 172.21.15.6:0/1100359056 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.231+0000 7f42ead9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f42ec09fe90 0x7f42ec1d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47384/0 (socket says 172.21.15.6:47384) 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42ea59c640 1 -- 172.21.15.6:0/1100359056 >> v1:172.21.15.6:6789/0 conn(0x7f42ec09d8e0 legacy=0x7f42ec097960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42ead9d640 1 -- 172.21.15.6:0/1100359056 >> v1:172.21.15.6:6790/0 conn(0x7f42ec09fe90 legacy=0x7f42ec1d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3905592404 0 0) 0x7f42ec098780 con 0x7f42ec09d8e0 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): my global_id is 4527 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42cc005740 con 0x7f42ec09d8e0 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3881745622 0 0) 0x7f42ec08d180 con 0x7f42ec09fe90 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): my global_id is 4487 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ec098780 con 0x7f42ec09fe90 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42e9d9b640 1 -- 172.21.15.6:0/1100359056 >> v1:172.21.15.203:6789/0 conn(0x7f42ec09e3d0 legacy=0x7f42ec098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 269667537 0 0) 0x7f42cc005740 con 0x7f42ec09d8e0 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42ec08d180 con 0x7f42ec09d8e0 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f42dc001ba0 con 0x7f42ec09d8e0 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): dump: 2024-04-25T21:40:50.245 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1889253295 0 0) 0x7f42ec098780 con 0x7f42ec09fe90 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42cc005740 con 0x7f42ec09fe90 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f42e4004d10 con 0x7f42ec09fe90 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:50.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): dump: 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3475045232 0 0) 0x7f42ec08cea0 con 0x7f42ec09e3d0 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:50.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient(hunting): my global_id is 94123 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ec098780 con 0x7f42ec09e3d0 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3964713944 0 0) 0x7f42ec08d180 con 0x7f42ec09d8e0 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 >> v1:172.21.15.6:6790/0 conn(0x7f42ec09fe90 legacy=0x7f42ec1d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 >> v1:172.21.15.203:6789/0 conn(0x7f42ec09e3d0 legacy=0x7f42ec098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 monclient: found mon.a 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ec08d460 con 0x7f42ec09d8e0 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 5 monclient: authenticate success, global_id 4527 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 10 monclient: _renew_subs 2024-04-25T21:40:50.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f42ec093e10 con 0x7f42ec09d8e0 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 10 monclient: _renew_subs 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f42ec08db60 con 0x7f42ec09d8e0 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f42dc003950 con 0x7f42ec09d8e0 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f42dc003ca0 con 0x7f42ec09d8e0 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 10 monclient: dump: 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:50.249 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42e8d99640 4 set_mon_vals no callback set 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f42dc003e40 con 0x7f42ec09d8e0 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f42dc0443e0 con 0x7f42ec09d8e0 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42ec062e50 con 0x7f42ec09d8e0 2024-04-25T21:40:50.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.235+0000 7f42e9d9b640 10 monclient: get_auth_request con 0x7f42cc038ef0 auth_method 0 2024-04-25T21:40:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.239+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f42dc0157b0 con 0x7f42ec09d8e0 2024-04-25T21:40:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.239+0000 7f42daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.239+0000 7f42daffd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:50.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f42f0dc4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-25T21:40:50.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f42f0dc4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-25T21:40:50.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f42f0dc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:50.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f42ec06c520 con 0x7f42ec09d8e0 2024-04-25T21:40:50.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f42daffd640 1 -- 172.21.15.6:0/1100359056 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f42dc019460 con 0x7f42ec09d8e0 2024-04-25T21:40:50.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f42daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-25T21:40:50.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f42daffd640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:50.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=1,health=12,kv=12,logm=60} crc {auth=487858573,config=3009771555,health=861809967,kv=684801865,logm=256123080}) 2024-04-25T21:40:50.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=13,logm=59} crc {auth=1799719789,config=1097430493,health=861809967,kv=377484006,logm=307212123}) 2024-04-25T21:40:50.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=1,health=12,kv=12,logm=60} crc {auth=487858573,config=3009771555,health=861809967,kv=684801865,logm=256123080}) 2024-04-25T21:40:50.469 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.463+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=12,logm=59} crc {auth=487858573,config=1097430493,health=861809967,kv=4075695600,logm=2004982807}) 2024-04-25T21:40:50.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 >> v1:172.21.15.203:6812/3164820986 conn(0x7f42cc038ef0 legacy=0x7f42cc03b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 10 monclient: shutdown 2024-04-25T21:40:50.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:50.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 >> v1:172.21.15.6:6789/0 conn(0x7f42ec09d8e0 legacy=0x7f42ec097960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:50.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 shutdown_connections 2024-04-25T21:40:50.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 >> 172.21.15.6:0/1100359056 conn(0x7f42ec0a2ee0 msgr2=0x7f42ec0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:50.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 shutdown_connections 2024-04-25T21:40:50.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f42f0dc4640 1 -- 172.21.15.6:0/1100359056 wait complete. 2024-04-25T21:40:50.472 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1529013763}) 2024-04-25T21:40:50.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=70351618}) 2024-04-25T21:40:50.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1529013763}) 2024-04-25T21:40:50.473 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3283151713}) 2024-04-25T21:40:50.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612875905}) 2024-04-25T21:40:50.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2680649783}) 2024-04-25T21:40:50.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612875905}) 2024-04-25T21:40:50.474 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.467+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=993078770}) 2024-04-25T21:40:50.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.471+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.471+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=2,mgr=7} crc {logm=650417384,mdsmap=3140514009,mgr=2133960809}) 2024-04-25T21:40:50.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.471+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=96,mdsmap=2,mgr=2} crc {logm=3173245605,mdsmap=3140514009,mgr=3439966934}) 2024-04-25T21:40:50.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.471+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.471+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=2,mgr=7} crc {logm=650417384,mdsmap=3140514009,mgr=2133960809}) 2024-04-25T21:40:50.476 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.471+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=4073628541,mdsmap=2840952586,mgr=3189685207}) 2024-04-25T21:40:50.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.475+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.475+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=49} crc {mgr=1747058506,mgr_metadata=579865471,mgrstat=2914016292,mon_config_key=1201469142,monmap=3140514009,osd_metadata=3973839794,osd_pg_creating=3251506115,osdmap=3019408518}) 2024-04-25T21:40:50.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.475+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=20,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=50} crc {mgr=1747058506,mgr_command_descs=3057339438,mgr_metadata=579865471,mgrstat=3930392220,mon_config_key=2785205056,monmap=3922754191,osd_metadata=3973839794,osd_pg_creating=3251506115,osdmap=418326110}) 2024-04-25T21:40:50.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.475+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-25T21:40:50.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.475+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_metadata=1,mgrstat=12,mon_config_key=19,monmap=2,osd_metadata=6,osd_pg_creating=1,osdmap=49} crc {mgr=1747058506,mgr_metadata=579865471,mgrstat=2914016292,mon_config_key=1201469142,monmap=3140514009,osd_metadata=3973839794,osd_pg_creating=3251506115,osdmap=3019408518}) 2024-04-25T21:40:50.480 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-04-25T21:40:50.475+0000 7f607f941640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=19,monmap=3,osd_metadata=6,osdmap=53} crc {mgr=1747058506,mgr_command_descs=3057339438,mgr_metadata=579865471,mgrstat=3930392220,mon_config_key=3474769134,monmap=3922754191,osd_metadata=3973839794,osdmap=4194259687}) 2024-04-25T21:40:50.486 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-25T21:40:51.487 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-25T21:40:51.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-25T21:40:51.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient: get_monmap_and_config 2024-04-25T21:40:51.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient: build_initial_monmap 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient: monmap: 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:51.621479+0000 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:51.621479+0000 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 1 Processor -- start 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 1 -- start start 2024-04-25T21:40:51.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient: init 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient(hunting): picked mon.noname-c con 0x7f72e80a3e80 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient(hunting): picked mon.noname-b con 0x7f72e80a4970 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient(hunting): picked mon.noname-a con 0x7f72e809bab0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e809c590 con 0x7f72e809bab0 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e809adf0 con 0x7f72e80a3e80 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e809bf20 con 0x7f72e80a4970 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72ece99640 10 monclient(hunting): authenticate will time out at 1010.531447s 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72e6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f72e809bab0 0x7f72e813f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34690/0 (socket says 172.21.15.6:34690) 2024-04-25T21:40:51.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.619+0000 7f72e6d9d640 1 -- 172.21.15.6:0/3826265567 learned_addr learned my addr 172.21.15.6:0/3826265567 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:51.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e6d9d640 1 -- 172.21.15.6:0/3826265567 >> v1:172.21.15.6:6789/0 conn(0x7f72e809bab0 legacy=0x7f72e813f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:51.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e5d9b640 1 -- 172.21.15.6:0/3826265567 >> v1:172.21.15.6:6790/0 conn(0x7f72e80a4970 legacy=0x7f72e813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:51.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361015966 0 0) 0x7f72e809c590 con 0x7f72e809bab0 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): _negotiate 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): my global_id is 4530 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d0003530 con 0x7f72e809bab0 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e659c640 1 -- 172.21.15.6:0/3826265567 >> v1:172.21.15.203:6789/0 conn(0x7f72e80a3e80 legacy=0x7f72e80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653453762 0 0) 0x7f72e809bf20 con 0x7f72e80a4970 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): _negotiate 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient(hunting): my global_id is 4490 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72e809c590 con 0x7f72e80a4970 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 606605588 0 0) 0x7f72d0003530 con 0x7f72e809bab0 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 >> v1:172.21.15.6:6790/0 conn(0x7f72e80a4970 legacy=0x7f72e813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:51.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 >> v1:172.21.15.203:6789/0 conn(0x7f72e80a3e80 legacy=0x7f72e80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: _finish_hunting 0 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 monclient: found mon.noname-a 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e806eaf0 con 0x7f72e809bab0 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: _finish_auth 0 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: _check_auth_tickets 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f72e0001c50 con 0x7f72e809bab0 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 5 monclient: authenticate success, global_id 4530 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: dump: 2024-04-25T21:40:51.627 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72e0001fa0 con 0x7f72e809bab0 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 4 set_mon_vals no callback set 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 1 -- 172.21.15.6:0/3826265567 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f72e0004dd0 con 0x7f72e809bab0 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:51.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72e559a640 10 monclient: dump: 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 10 monclient: shutdown 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 1 -- 172.21.15.6:0/3826265567 >> v1:172.21.15.6:6789/0 conn(0x7f72e809bab0 legacy=0x7f72e813f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:51.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 1 -- 172.21.15.6:0/3826265567 shutdown_connections 2024-04-25T21:40:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 1 -- 172.21.15.6:0/3826265567 >> 172.21.15.6:0/3826265567 conn(0x7f72e80a5f80 msgr2=0x7f72e80a6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 1 -- 172.21.15.6:0/3826265567 shutdown_connections 2024-04-25T21:40:51.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.623+0000 7f72ece99640 1 -- 172.21.15.6:0/3826265567 wait complete. 2024-04-25T21:40:51.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient: build_initial_monmap 2024-04-25T21:40:51.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient: monmap: 2024-04-25T21:40:51.634 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:51.634 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:51.634 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:51.634 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 1 Processor -- start 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 1 -- start start 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient: init 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient(hunting): picked mon.noname-c con 0x7f72e809bab0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient(hunting): picked mon.noname-b con 0x7f72e80a3e80 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient(hunting): picked mon.noname-a con 0x7f72e80a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e81be950 con 0x7f72e80a4970 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e81a9bf0 con 0x7f72e80a3e80 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72e80a5c80 con 0x7f72e809bab0 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72ece99640 10 monclient(hunting): authenticate will time out at 1010.537030s 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72e659c640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f72e809bab0 0x7f72e81bde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47388/0 (socket says 172.21.15.6:47388) 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72e6d9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f72e80a4970 0x7f72e81d3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34694/0 (socket says 172.21.15.6:34694) 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72e659c640 1 -- 172.21.15.6:0/1147562024 learned_addr learned my addr 172.21.15.6:0/1147562024 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72e659c640 1 -- 172.21.15.6:0/1147562024 >> v1:172.21.15.6:6790/0 conn(0x7f72e809bab0 legacy=0x7f72e81bde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72e6d9d640 1 -- 172.21.15.6:0/1147562024 >> v1:172.21.15.6:6789/0 conn(0x7f72e80a4970 legacy=0x7f72e81d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 959059781 0 0) 0x7f72e81be950 con 0x7f72e80a4970 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:51.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): my global_id is 4533 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72c8003530 con 0x7f72e80a4970 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982399909 0 0) 0x7f72e80a5c80 con 0x7f72e809bab0 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): my global_id is 4493 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72e81be950 con 0x7f72e809bab0 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72e5d9b640 1 -- 172.21.15.6:0/1147562024 >> v1:172.21.15.203:6789/0 conn(0x7f72e80a3e80 legacy=0x7f72e81d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 77905727 0 0) 0x7f72c8003530 con 0x7f72e80a4970 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72e80a5c80 con 0x7f72e80a4970 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f72e0004d10 con 0x7f72e80a4970 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:51.640 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2157866851 0 0) 0x7f72e81be950 con 0x7f72e809bab0 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f72c8003530 con 0x7f72e809bab0 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f72dc001c50 con 0x7f72e809bab0 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:51.641 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.627+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2086840226 0 0) 0x7f72e81a9bf0 con 0x7f72e80a3e80 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient(hunting): my global_id is 94129 2024-04-25T21:40:51.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72e81be950 con 0x7f72e80a3e80 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 833179856 0 0) 0x7f72e80a5c80 con 0x7f72e80a4970 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 >> v1:172.21.15.6:6790/0 conn(0x7f72e809bab0 legacy=0x7f72e81bde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 >> v1:172.21.15.203:6789/0 conn(0x7f72e80a3e80 legacy=0x7f72e81d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 monclient: found mon.a 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e81cbac0 con 0x7f72e80a4970 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 5 monclient: authenticate success, global_id 4533 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 10 monclient: _renew_subs 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:51.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f72e81a9310 con 0x7f72e80a4970 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 10 monclient: _renew_subs 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f72e81aad60 con 0x7f72e80a4970 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f72e0005390 con 0x7f72e80a4970 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f72e00056e0 con 0x7f72e80a4970 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72e4d99640 4 set_mon_vals no callback set 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 10 monclient: dump: 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72cc005090 con 0x7f72e80a4970 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f72e0010480 con 0x7f72e80a4970 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f72e0044580 con 0x7f72e80a4970 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.631+0000 7f72e659c640 10 monclient: get_auth_request con 0x7f72c8036c00 auth_method 0 2024-04-25T21:40:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.635+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f72e0015af0 con 0x7f72e80a4970 2024-04-25T21:40:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.635+0000 7f72d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.635+0000 7f72d6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.855+0000 7f72ece99640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-25T21:40:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.855+0000 7f72ece99640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.855+0000 7f72ece99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:51.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.855+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f72cc005380 con 0x7f72e80a4970 2024-04-25T21:40:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.855+0000 7f72d6ffd640 1 -- 172.21.15.6:0/1147562024 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 2396355542) 0x7f72e00153f0 con 0x7f72e80a4970 2024-04-25T21:40:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.855+0000 7f72d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-25T21:40:51.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.855+0000 7f72d6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:51.857 INFO:teuthology.orchestra.run.smithi006.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","modified":"2024-04-25T21:39:14.743126Z","created":"2024-04-25T21:39:14.743126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 >> v1:172.21.15.203:6812/3164820986 conn(0x7f72c8036c00 legacy=0x7f72c80390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 10 monclient: shutdown 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 >> v1:172.21.15.6:6789/0 conn(0x7f72e80a4970 legacy=0x7f72e81d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 shutdown_connections 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 >> 172.21.15.6:0/1147562024 conn(0x7f72e80a5f80 msgr2=0x7f72e809ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 shutdown_connections 2024-04-25T21:40:51.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:51.859+0000 7f72ece99640 1 -- 172.21.15.6:0/1147562024 wait complete. 2024-04-25T21:40:51.875 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-25T21:40:51.875 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T21:40:51.884 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T21:40:51.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.003+0000 7f25d41ea640 10 monclient: get_monmap_and_config 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.003+0000 7f25d41ea640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.003+0000 7f25d41ea640 10 monclient: monmap: 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:52.006500+0000 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:52.006500+0000 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.005 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:52.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:52.006 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.003+0000 7f25d41ea640 1 Processor -- start 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.003+0000 7f25d41ea640 1 -- start start 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.003+0000 7f25d41ea640 10 monclient: init 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient(hunting): picked mon.noname-a con 0x7f25cc09e860 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient(hunting): picked mon.noname-b con 0x7f25cc0a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient(hunting): picked mon.noname-c con 0x7f25cc0a1680 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25cc09c130 con 0x7f25cc09e860 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25cc099520 con 0x7f25cc0a1680 2024-04-25T21:40:52.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25cc099b10 con 0x7f25cc0a1270 2024-04-25T21:40:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient(hunting): authenticate will time out at 1010.916513s 2024-04-25T21:40:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d1f86640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25cc09e860 0x7f25cc0a0b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34702/0 (socket says 172.21.15.6:34702) 2024-04-25T21:40:52.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d1f86640 1 -- 172.21.15.6:0/1993757863 learned_addr learned my addr 172.21.15.6:0/1993757863 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d1f86640 1 -- 172.21.15.6:0/1993757863 >> v1:172.21.15.6:6789/0 conn(0x7f25cc09e860 legacy=0x7f25cc0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d1785640 1 -- 172.21.15.6:0/1993757863 >> v1:172.21.15.6:6790/0 conn(0x7f25cc0a1270 legacy=0x7f25cc139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 756372733 0 0) 0x7f25c80017e0 con 0x7f25cc09e860 2024-04-25T21:40:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): my global_id is 4536 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b8003530 con 0x7f25cc09e860 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382937955 0 0) 0x7f25cc099b10 con 0x7f25cc0a1270 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient(hunting): my global_id is 4496 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25c80017e0 con 0x7f25cc0a1270 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d2787640 1 -- 172.21.15.6:0/1993757863 >> v1:172.21.15.203:6789/0 conn(0x7f25cc0a1680 legacy=0x7f25cc13d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1958678266 0 0) 0x7f25b8003530 con 0x7f25cc09e860 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 >> v1:172.21.15.6:6790/0 conn(0x7f25cc0a1270 legacy=0x7f25cc139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 >> v1:172.21.15.203:6789/0 conn(0x7f25cc0a1680 legacy=0x7f25cc13d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 monclient: found mon.noname-a 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25cc09a0d0 con 0x7f25cc09e860 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f25c8001ee0 con 0x7f25cc09e860 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: dump: 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.012 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 5 monclient: authenticate success, global_id 4536 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25c8004c30 con 0x7f25cc09e860 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 1 -- 172.21.15.6:0/1993757863 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f25c8004f80 con 0x7f25cc09e860 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d0f84640 10 monclient: dump: 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.013 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.007+0000 7f25d41ea640 4 set_mon_vals no callback set 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient: shutdown 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- 172.21.15.6:0/1993757863 >> v1:172.21.15.6:6789/0 conn(0x7f25cc09e860 legacy=0x7f25cc0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- 172.21.15.6:0/1993757863 shutdown_connections 2024-04-25T21:40:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- 172.21.15.6:0/1993757863 >> 172.21.15.6:0/1993757863 conn(0x7f25cc0a1c50 msgr2=0x7f25cc0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- 172.21.15.6:0/1993757863 shutdown_connections 2024-04-25T21:40:52.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- 172.21.15.6:0/1993757863 wait complete. 2024-04-25T21:40:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient: monmap: 2024-04-25T21:40:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:52.019 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 Processor -- start 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- start start 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient: init 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient(hunting): picked mon.noname-b con 0x7f25cc09e860 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient(hunting): picked mon.noname-c con 0x7f25cc0a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient(hunting): picked mon.noname-a con 0x7f25cc0a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25cc072b40 con 0x7f25cc0a1680 2024-04-25T21:40:52.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25cc0981f0 con 0x7f25cc09e860 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25cc09e040 con 0x7f25cc0a1270 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d41ea640 10 monclient(hunting): authenticate will time out at 1010.921922s 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d2787640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25cc0a1680 0x7f25cc1d4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34708/0 (socket says 172.21.15.6:34708) 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d1785640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f25cc0a1270 0x7f25cc1d0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47398/0 (socket says 172.21.15.6:47398) 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.011+0000 7f25d2787640 1 -- 172.21.15.6:0/2450275501 learned_addr learned my addr 172.21.15.6:0/2450275501 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d1785640 1 -- 172.21.15.6:0/2450275501 >> v1:172.21.15.6:6790/0 conn(0x7f25cc0a1270 legacy=0x7f25cc1d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d2787640 1 -- 172.21.15.6:0/2450275501 >> v1:172.21.15.6:6789/0 conn(0x7f25cc0a1680 legacy=0x7f25cc1d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782288869 0 0) 0x7f25cc09e040 con 0x7f25cc0a1270 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): my global_id is 4499 2024-04-25T21:40:52.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25b0003530 con 0x7f25cc0a1270 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1406596869 0 0) 0x7f25cc072b40 con 0x7f25cc0a1680 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): my global_id is 4539 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25cc09e040 con 0x7f25cc0a1680 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d1f86640 1 -- 172.21.15.6:0/2450275501 >> v1:172.21.15.203:6789/0 conn(0x7f25cc09e860 legacy=0x7f25cc13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 433440046 0 0) 0x7f25b0003530 con 0x7f25cc0a1270 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25cc072b40 con 0x7f25cc0a1270 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f25bc001630 con 0x7f25cc0a1270 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2370422289 0 0) 0x7f25cc09e040 con 0x7f25cc0a1680 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f25b0003530 con 0x7f25cc0a1680 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f25c4001ba0 con 0x7f25cc0a1680 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3548826124 0 0) 0x7f25cc0981f0 con 0x7f25cc09e860 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient(hunting): my global_id is 94135 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25cc09e040 con 0x7f25cc09e860 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3203398284 0 0) 0x7f25cc072b40 con 0x7f25cc0a1270 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 >> v1:172.21.15.203:6789/0 conn(0x7f25cc09e860 legacy=0x7f25cc13d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 >> v1:172.21.15.6:6789/0 conn(0x7f25cc0a1680 legacy=0x7f25cc1d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 monclient: found mon.c 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25cc09dd10 con 0x7f25cc0a1270 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 5 monclient: authenticate success, global_id 4499 2024-04-25T21:40:52.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 10 monclient: _renew_subs 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f25cc091fc0 con 0x7f25cc0a1270 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 10 monclient: _renew_subs 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f25cc091410 con 0x7f25cc0a1270 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25cc062e50 con 0x7f25cc0a1270 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f25bc003950 con 0x7f25cc0a1270 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f25bc003ca0 con 0x7f25cc0a1270 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c3fff640 4 set_mon_vals no callback set 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 10 monclient: dump: 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f25bc003e40 con 0x7f25cc0a1270 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f25bc0442b0 con 0x7f25cc0a1270 2024-04-25T21:40:52.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.015+0000 7f25d1f86640 10 monclient: get_auth_request con 0x7f25b0036ce0 auth_method 0 2024-04-25T21:40:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.019+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f25bc015690 con 0x7f25cc0a1270 2024-04-25T21:40:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.019+0000 7f25c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.019+0000 7f25c27fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:52.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.179+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f25cc06b2e0 con 0x7f25b0036ce0 2024-04-25T21:40:52.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25c27fc640 1 -- 172.21.15.6:0/2450275501 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33735 (unknown 2965378022 0 3068148926) 0x7f25cc06b2e0 con 0x7f25b0036ce0 2024-04-25T21:40:52.183 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:52.184 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 >> v1:172.21.15.203:6812/3164820986 conn(0x7f25b0036ce0 legacy=0x7f25b00391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 10 monclient: shutdown 2024-04-25T21:40:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 >> v1:172.21.15.6:6790/0 conn(0x7f25cc0a1270 legacy=0x7f25cc1d0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 shutdown_connections 2024-04-25T21:40:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 >> 172.21.15.6:0/2450275501 conn(0x7f25cc0a1c50 msgr2=0x7f25cc098760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 shutdown_connections 2024-04-25T21:40:52.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.183+0000 7f25d41ea640 1 -- 172.21.15.6:0/2450275501 wait complete. 2024-04-25T21:40:52.200 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-04-25T21:40:51.530162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168632,"kb_used_data":2336,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566062408,"statfs":{"total":579820584960,"available":579647905792,"internally_reserved":0,"allocated":2392064,"data_stored":1513010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003221"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.615368+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:35.615368+0000","last_peered":"2024-04-25T21:40:35.615368+0000","last_clean":"2024-04-25T21:40:35.615368+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:35.615368+0000","last_undegraded":"2024-04-25T21:40:35.615368+0000","last_fullsized":"2024-04-25T21:40:35.615368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.612967+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:35.612967+0000","last_peered":"2024-04-25T21:40:35.612967+0000","last_clean":"2024-04-25T21:40:35.612967+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:35.612967+0000","last_undegraded":"2024-04-25T21:40:35.612967+0000","last_fullsized":"2024-04-25T21:40:35.612967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.616412+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:35.616412+0000","last_peered":"2024-04-25T21:40:35.616412+0000","last_clean":"2024-04-25T21:40:35.616412+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:35.616412+0000","last_undegraded":"2024-04-25T21:40:35.616412+0000","last_fullsized":"2024-04-25T21:40:35.616412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614816+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:35.614816+0000","last_peered":"2024-04-25T21:40:35.614816+0000","last_clean":"2024-04-25T21:40:35.614816+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:35.614816+0000","last_undegraded":"2024-04-25T21:40:35.614816+0000","last_fullsized":"2024-04-25T21:40:35.614816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.617201+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:35.617201+0000","last_peered":"2024-04-25T21:40:35.617201+0000","last_clean":"2024-04-25T21:40:35.617201+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:35.617201+0000","last_undegraded":"2024-04-25T21:40:35.617201+0000","last_fullsized":"2024-04-25T21:40:35.617201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:34.534127+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:34.534127+0000","last_peered":"2024-04-25T21:40:34.534127+0000","last_clean":"2024-04-25T21:40:34.534127+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:34.534127+0000","last_undegraded":"2024-04-25T21:40:34.534127+0000","last_fullsized":"2024-04-25T21:40:34.534127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613555+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:35.613555+0000","last_peered":"2024-04-25T21:40:35.613555+0000","last_clean":"2024-04-25T21:40:35.613555+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:35.613555+0000","last_undegraded":"2024-04-25T21:40:35.613555+0000","last_fullsized":"2024-04-25T21:40:35.613555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614625+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:35.614625+0000","last_peered":"2024-04-25T21:40:35.614625+0000","last_clean":"2024-04-25T21:40:35.614625+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:35.614625+0000","last_undegraded":"2024-04-25T21:40:35.614625+0000","last_fullsized":"2024-04-25T21:40:35.614625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613838+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:35.613838+0000","last_peered":"2024-04-25T21:40:35.613838+0000","last_clean":"2024-04-25T21:40:35.613838+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:35.613838+0000","last_undegraded":"2024-04-25T21:40:35.613838+0000","last_fullsized":"2024-04-25T21:40:35.613838+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:52.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.319+0000 7fa274667640 10 monclient: get_monmap_and_config 2024-04-25T21:40:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.319+0000 7fa274667640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.319+0000 7fa274667640 10 monclient: monmap: 2024-04-25T21:40:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:52.322672+0000 2024-04-25T21:40:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:52.322672+0000 2024-04-25T21:40:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:52.322 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:52.322 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.319+0000 7fa274667640 1 Processor -- start 2024-04-25T21:40:52.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.319+0000 7fa274667640 1 -- start start 2024-04-25T21:40:52.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.319+0000 7fa274667640 10 monclient: init 2024-04-25T21:40:52.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient(hunting): picked mon.noname-a con 0x7fa26c0a0b60 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient(hunting): picked mon.noname-c con 0x7fa26c09ed30 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient(hunting): picked mon.noname-b con 0x7fa26c09f180 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c09aa70 con 0x7fa26c0a0b60 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c072c40 con 0x7fa26c09ed30 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c09bbf0 con 0x7fa26c09f180 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient(hunting): authenticate will time out at 1011.232503s 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa272c04640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa26c09f180 0x7fa26c13c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40818/0 (socket says 172.21.15.6:40818) 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa272403640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa26c0a0b60 0x7fa26c09a360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55562/0 (socket says 172.21.15.6:55562) 2024-04-25T21:40:52.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa272c04640 1 -- 172.21.15.6:0/2769509710 learned_addr learned my addr 172.21.15.6:0/2769509710 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa272c04640 1 -- 172.21.15.6:0/2769509710 >> v1:172.21.15.6:6790/0 conn(0x7fa26c09f180 legacy=0x7fa26c13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa272403640 1 -- 172.21.15.6:0/2769509710 >> v1:172.21.15.6:6789/0 conn(0x7fa26c0a0b60 legacy=0x7fa26c09a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197833695 0 0) 0x7fa26c09aa70 con 0x7fa26c0a0b60 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): my global_id is 4542 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa258003530 con 0x7fa26c0a0b60 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844600401 0 0) 0x7fa26c09bbf0 con 0x7fa26c09f180 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient(hunting): my global_id is 4502 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa26c09aa70 con 0x7fa26c09f180 2024-04-25T21:40:52.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271c02640 1 -- 172.21.15.6:0/2769509710 >> v1:172.21.15.203:6789/0 conn(0x7fa26c09ed30 legacy=0x7fa26c138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3636842097 0 0) 0x7fa258003530 con 0x7fa26c0a0b60 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 >> v1:172.21.15.6:6790/0 conn(0x7fa26c09f180 legacy=0x7fa26c13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 >> v1:172.21.15.203:6789/0 conn(0x7fa26c09ed30 legacy=0x7fa26c138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 monclient: found mon.noname-a 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c09bed0 con 0x7fa26c0a0b60 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa264001c50 con 0x7fa26c0a0b60 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 5 monclient: authenticate success, global_id 4542 2024-04-25T21:40:52.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: dump: 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa264001fa0 con 0x7fa26c0a0b60 2024-04-25T21:40:52.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 1 -- 172.21.15.6:0/2769509710 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa264004d10 con 0x7fa26c0a0b60 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa271401640 10 monclient: dump: 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 4 set_mon_vals no callback set 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 10 monclient: shutdown 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.323+0000 7fa274667640 1 -- 172.21.15.6:0/2769509710 >> v1:172.21.15.6:6789/0 conn(0x7fa26c0a0b60 legacy=0x7fa26c09a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- 172.21.15.6:0/2769509710 shutdown_connections 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- 172.21.15.6:0/2769509710 >> 172.21.15.6:0/2769509710 conn(0x7fa26c0a1c50 msgr2=0x7fa26c0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- 172.21.15.6:0/2769509710 shutdown_connections 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- 172.21.15.6:0/2769509710 wait complete. 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient: monmap: 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 Processor -- start 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- start start 2024-04-25T21:40:52.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient: init 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient(hunting): picked mon.noname-a con 0x7fa26c09ed30 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient(hunting): picked mon.noname-c con 0x7fa26c09f180 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient(hunting): picked mon.noname-b con 0x7fa26c0a0b60 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c12e3a0 con 0x7fa26c09ed30 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c12cb10 con 0x7fa26c0a0b60 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c09b730 con 0x7fa26c09f180 2024-04-25T21:40:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa274667640 10 monclient(hunting): authenticate will time out at 1011.237491s 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa272403640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa26c09ed30 0x7fa26c12ad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55566/0 (socket says 172.21.15.6:55566) 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa272403640 1 -- 172.21.15.6:0/3606825864 learned_addr learned my addr 172.21.15.6:0/3606825864 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa271c02640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa26c09f180 0x7fa26c12b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40834/0 (socket says 172.21.15.6:40834) 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa272403640 1 -- 172.21.15.6:0/3606825864 >> v1:172.21.15.6:6789/0 conn(0x7fa26c09ed30 legacy=0x7fa26c12ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa271c02640 1 -- 172.21.15.6:0/3606825864 >> v1:172.21.15.6:6790/0 conn(0x7fa26c09f180 legacy=0x7fa26c12b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788267303 0 0) 0x7fa26c12e3a0 con 0x7fa26c09ed30 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): my global_id is 4545 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa250003530 con 0x7fa26c09ed30 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718831500 0 0) 0x7fa26c09b730 con 0x7fa26c09f180 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 10 monclient(hunting): my global_id is 4505 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.327+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa26c12e3a0 con 0x7fa26c09f180 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa272c04640 1 -- 172.21.15.6:0/3606825864 >> v1:172.21.15.203:6789/0 conn(0x7fa26c0a0b60 legacy=0x7fa26c1cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 54903392 0 0) 0x7fa250003530 con 0x7fa26c09ed30 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa26c09b730 con 0x7fa26c09ed30 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa264001c10 con 0x7fa26c09ed30 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 157795708 0 0) 0x7fa26c12e3a0 con 0x7fa26c09f180 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa250003530 con 0x7fa26c09f180 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa25c001b00 con 0x7fa26c09f180 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3132435269 0 0) 0x7fa26c09b730 con 0x7fa26c09ed30 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 >> v1:172.21.15.6:6790/0 conn(0x7fa26c09f180 legacy=0x7fa26c12b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 >> v1:172.21.15.203:6789/0 conn(0x7fa26c0a0b60 legacy=0x7fa26c1cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 monclient: found mon.a 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c1ba400 con 0x7fa26c09ed30 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 5 monclient: authenticate success, global_id 4545 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 10 monclient: _renew_subs 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa26c12e960 con 0x7fa26c09ed30 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 10 monclient: _renew_subs 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa26c12bd20 con 0x7fa26c09ed30 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa264001fa0 con 0x7fa26c09ed30 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa264005460 con 0x7fa26c09ed30 2024-04-25T21:40:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa270c00640 4 set_mon_vals no callback set 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 10 monclient: dump: 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.342 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa254005090 con 0x7fa26c09ed30 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fa264010200 con 0x7fa26c09ed30 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fa2640447a0 con 0x7fa26c09ed30 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.331+0000 7fa271c02640 10 monclient: get_auth_request con 0x7fa250036ac0 auth_method 0 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.335+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fa264015b20 con 0x7fa26c09ed30 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.335+0000 7fa262ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.335+0000 7fa262ffd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:52.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.487+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa254000ba0 con 0x7fa250036ac0 2024-04-25T21:40:52.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.487+0000 7fa262ffd640 1 -- 172.21.15.6:0/3606825864 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33735 (unknown 2965378022 0 3068148926) 0x7fa254000ba0 con 0x7fa250036ac0 2024-04-25T21:40:52.490 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 >> v1:172.21.15.203:6812/3164820986 conn(0x7fa250036ac0 legacy=0x7fa250038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 10 monclient: shutdown 2024-04-25T21:40:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 >> v1:172.21.15.6:6789/0 conn(0x7fa26c09ed30 legacy=0x7fa26c12ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 shutdown_connections 2024-04-25T21:40:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 >> 172.21.15.6:0/3606825864 conn(0x7fa26c0a1c50 msgr2=0x7fa26c1b93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 shutdown_connections 2024-04-25T21:40:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.491+0000 7fa274667640 1 -- 172.21.15.6:0/3606825864 wait complete. 2024-04-25T21:40:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-04-25T21:40:51.530162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168632,"kb_used_data":2336,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566062408,"statfs":{"total":579820584960,"available":579647905792,"internally_reserved":0,"allocated":2392064,"data_stored":1513010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003221"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.615368+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:35.615368+0000","last_peered":"2024-04-25T21:40:35.615368+0000","last_clean":"2024-04-25T21:40:35.615368+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:35.615368+0000","last_undegraded":"2024-04-25T21:40:35.615368+0000","last_fullsized":"2024-04-25T21:40:35.615368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.612967+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:35.612967+0000","last_peered":"2024-04-25T21:40:35.612967+0000","last_clean":"2024-04-25T21:40:35.612967+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:35.612967+0000","last_undegraded":"2024-04-25T21:40:35.612967+0000","last_fullsized":"2024-04-25T21:40:35.612967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.616412+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:35.616412+0000","last_peered":"2024-04-25T21:40:35.616412+0000","last_clean":"2024-04-25T21:40:35.616412+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:35.616412+0000","last_undegraded":"2024-04-25T21:40:35.616412+0000","last_fullsized":"2024-04-25T21:40:35.616412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614816+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:35.614816+0000","last_peered":"2024-04-25T21:40:35.614816+0000","last_clean":"2024-04-25T21:40:35.614816+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:35.614816+0000","last_undegraded":"2024-04-25T21:40:35.614816+0000","last_fullsized":"2024-04-25T21:40:35.614816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.617201+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:35.617201+0000","last_peered":"2024-04-25T21:40:35.617201+0000","last_clean":"2024-04-25T21:40:35.617201+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:35.617201+0000","last_undegraded":"2024-04-25T21:40:35.617201+0000","last_fullsized":"2024-04-25T21:40:35.617201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:34.534127+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:34.534127+0000","last_peered":"2024-04-25T21:40:34.534127+0000","last_clean":"2024-04-25T21:40:34.534127+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:34.534127+0000","last_undegraded":"2024-04-25T21:40:34.534127+0000","last_fullsized":"2024-04-25T21:40:34.534127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613555+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:35.613555+0000","last_peered":"2024-04-25T21:40:35.613555+0000","last_clean":"2024-04-25T21:40:35.613555+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:35.613555+0000","last_undegraded":"2024-04-25T21:40:35.613555+0000","last_fullsized":"2024-04-25T21:40:35.613555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614625+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:35.614625+0000","last_peered":"2024-04-25T21:40:35.614625+0000","last_clean":"2024-04-25T21:40:35.614625+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:35.614625+0000","last_undegraded":"2024-04-25T21:40:35.614625+0000","last_fullsized":"2024-04-25T21:40:35.614625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613838+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:35.613838+0000","last_peered":"2024-04-25T21:40:35.613838+0000","last_clean":"2024-04-25T21:40:35.613838+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:35.613838+0000","last_undegraded":"2024-04-25T21:40:35.613838+0000","last_fullsized":"2024-04-25T21:40:35.613838+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:52.507 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T21:40:52.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.623+0000 7fbaeed20640 10 monclient: get_monmap_and_config 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.623+0000 7fbaeed20640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.623+0000 7fbaeed20640 10 monclient: monmap: 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:52.626483+0000 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:52.626483+0000 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.625 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.626 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.626 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:52.626 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:52.626 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.623+0000 7fbaeed20640 1 Processor -- start 2024-04-25T21:40:52.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.623+0000 7fbaeed20640 1 -- start start 2024-04-25T21:40:52.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.623+0000 7fbaeed20640 10 monclient: init 2024-04-25T21:40:52.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:52.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient(hunting): picked mon.noname-a con 0x7fbae80a2c10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient(hunting): picked mon.noname-b con 0x7fbae8139e50 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient(hunting): picked mon.noname-c con 0x7fbae813d960 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a0e40 con 0x7fbae80a2c10 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a1400 con 0x7fbae813d960 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a3700 con 0x7fbae8139e50 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient(hunting): authenticate will time out at 1011.536429s 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaecabc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbae80a2c10 0x7fbae80a2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55568/0 (socket says 172.21.15.6:55568) 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae7fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbae8139e50 0x7fbae813c240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40838/0 (socket says 172.21.15.6:40838) 2024-04-25T21:40:52.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaecabc640 1 -- 172.21.15.6:0/2370008183 learned_addr learned my addr 172.21.15.6:0/2370008183 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaecabc640 1 -- 172.21.15.6:0/2370008183 >> v1:172.21.15.6:6789/0 conn(0x7fbae80a2c10 legacy=0x7fbae80a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae7fff640 1 -- 172.21.15.6:0/2370008183 >> v1:172.21.15.6:6790/0 conn(0x7fbae8139e50 legacy=0x7fbae813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069728815 0 0) 0x7fbae80a0e40 con 0x7fbae80a2c10 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): my global_id is 4548 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad0003530 con 0x7fbae80a2c10 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098381352 0 0) 0x7fbae80a3700 con 0x7fbae8139e50 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient(hunting): my global_id is 4508 2024-04-25T21:40:52.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae80a0e40 con 0x7fbae8139e50 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaed2bd640 1 -- 172.21.15.6:0/2370008183 >> v1:172.21.15.203:6789/0 conn(0x7fbae813d960 legacy=0x7fbae813fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1649200708 0 0) 0x7fbad0003530 con 0x7fbae80a2c10 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 >> v1:172.21.15.6:6790/0 conn(0x7fbae8139e50 legacy=0x7fbae813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 >> v1:172.21.15.203:6789/0 conn(0x7fbae813d960 legacy=0x7fbae813fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 monclient: found mon.noname-a 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae806e9f0 con 0x7fbae80a2c10 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbad8001c50 con 0x7fbae80a2c10 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 5 monclient: authenticate success, global_id 4548 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: dump: 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.634 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbad8001fa0 con 0x7fbae80a2c10 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 1 -- 172.21.15.6:0/2370008183 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbad8004dd0 con 0x7fbae80a2c10 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbae77fe640 10 monclient: dump: 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.635 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 4 set_mon_vals no callback set 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 10 monclient: shutdown 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.627+0000 7fbaeed20640 1 -- 172.21.15.6:0/2370008183 >> v1:172.21.15.6:6789/0 conn(0x7fbae80a2c10 legacy=0x7fbae80a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- 172.21.15.6:0/2370008183 shutdown_connections 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- 172.21.15.6:0/2370008183 >> 172.21.15.6:0/2370008183 conn(0x7fbae80a5e10 msgr2=0x7fbae80a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:52.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- 172.21.15.6:0/2370008183 shutdown_connections 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- 172.21.15.6:0/2370008183 wait complete. 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient: monmap: 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 Processor -- start 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- start start 2024-04-25T21:40:52.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient: init 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient(hunting): picked mon.noname-c con 0x7fbae80a2c10 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient(hunting): picked mon.noname-a con 0x7fbae8139e50 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient(hunting): picked mon.noname-b con 0x7fbae813d960 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae81392a0 con 0x7fbae8139e50 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae8124530 con 0x7fbae813d960 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae80a5930 con 0x7fbae80a2c10 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaeed20640 10 monclient(hunting): authenticate will time out at 1011.541449s 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae7fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbae8139e50 0x7fbae81d0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55580/0 (socket says 172.21.15.6:55580) 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaecabc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbae80a2c10 0x7fbae8138b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40852/0 (socket says 172.21.15.6:40852) 2024-04-25T21:40:52.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae7fff640 1 -- 172.21.15.6:0/974451428 learned_addr learned my addr 172.21.15.6:0/974451428 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaecabc640 1 -- 172.21.15.6:0/974451428 >> v1:172.21.15.6:6790/0 conn(0x7fbae80a2c10 legacy=0x7fbae8138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae7fff640 1 -- 172.21.15.6:0/974451428 >> v1:172.21.15.6:6789/0 conn(0x7fbae8139e50 legacy=0x7fbae81d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792840812 0 0) 0x7fbae80a5930 con 0x7fbae80a2c10 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): my global_id is 4511 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbad4003530 con 0x7fbae80a2c10 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054681067 0 0) 0x7fbae81392a0 con 0x7fbae8139e50 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 10 monclient(hunting): my global_id is 4551 2024-04-25T21:40:52.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae80a5930 con 0x7fbae8139e50 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.631+0000 7fbaed2bd640 1 -- 172.21.15.6:0/974451428 >> v1:172.21.15.203:6789/0 conn(0x7fbae813d960 legacy=0x7fbae81d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2565721354 0 0) 0x7fbad4003530 con 0x7fbae80a2c10 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbae81392a0 con 0x7fbae80a2c10 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbad8004d10 con 0x7fbae80a2c10 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): dump: 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.643 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 332534976 0 0) 0x7fbae80a5930 con 0x7fbae8139e50 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbad4003530 con 0x7fbae8139e50 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbadc001c40 con 0x7fbae8139e50 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): dump: 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.644 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477725454 0 0) 0x7fbae8124530 con 0x7fbae813d960 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient(hunting): my global_id is 94147 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbae80a5930 con 0x7fbae813d960 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2489762143 0 0) 0x7fbae81392a0 con 0x7fbae80a2c10 2024-04-25T21:40:52.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 >> v1:172.21.15.203:6789/0 conn(0x7fbae813d960 legacy=0x7fbae81d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 >> v1:172.21.15.6:6789/0 conn(0x7fbae8139e50 legacy=0x7fbae81d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 monclient: found mon.c 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae813cf00 con 0x7fbae80a2c10 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 5 monclient: authenticate success, global_id 4511 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 10 monclient: _renew_subs 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbae8122240 con 0x7fbae80a2c10 2024-04-25T21:40:52.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 10 monclient: _renew_subs 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbae8122b20 con 0x7fbae80a2c10 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbad8001f20 con 0x7fbae80a2c10 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbad80055e0 con 0x7fbae80a2c10 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae6ffd640 4 set_mon_vals no callback set 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 10 monclient: dump: 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.647 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fbad8010380 con 0x7fbae80a2c10 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac8005090 con 0x7fbae80a2c10 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fbad80446d0 con 0x7fbae80a2c10 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.635+0000 7fbae7fff640 10 monclient: get_auth_request con 0x7fbad4036c00 auth_method 0 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.639+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fbad8015bd0 con 0x7fbae80a2c10 2024-04-25T21:40:52.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.639+0000 7fbae57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.639+0000 7fbae57fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.799+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbac8000ba0 con 0x7fbad4036c00 2024-04-25T21:40:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbae57fa640 1 -- 172.21.15.6:0/974451428 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33735 (unknown 2965378022 0 3068148926) 0x7fbac8000ba0 con 0x7fbad4036c00 2024-04-25T21:40:52.803 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:52.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 >> v1:172.21.15.203:6812/3164820986 conn(0x7fbad4036c00 legacy=0x7fbad40390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 10 monclient: shutdown 2024-04-25T21:40:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 >> v1:172.21.15.6:6790/0 conn(0x7fbae80a2c10 legacy=0x7fbae8138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 shutdown_connections 2024-04-25T21:40:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 >> 172.21.15.6:0/974451428 conn(0x7fbae80a5e10 msgr2=0x7fbae80a8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 shutdown_connections 2024-04-25T21:40:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.803+0000 7fbaeed20640 1 -- 172.21.15.6:0/974451428 wait complete. 2024-04-25T21:40:52.821 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-04-25T21:40:51.530162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168632,"kb_used_data":2336,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566062408,"statfs":{"total":579820584960,"available":579647905792,"internally_reserved":0,"allocated":2392064,"data_stored":1513010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003221"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.615368+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:35.615368+0000","last_peered":"2024-04-25T21:40:35.615368+0000","last_clean":"2024-04-25T21:40:35.615368+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:35.615368+0000","last_undegraded":"2024-04-25T21:40:35.615368+0000","last_fullsized":"2024-04-25T21:40:35.615368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.612967+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:35.612967+0000","last_peered":"2024-04-25T21:40:35.612967+0000","last_clean":"2024-04-25T21:40:35.612967+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:35.612967+0000","last_undegraded":"2024-04-25T21:40:35.612967+0000","last_fullsized":"2024-04-25T21:40:35.612967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.616412+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:35.616412+0000","last_peered":"2024-04-25T21:40:35.616412+0000","last_clean":"2024-04-25T21:40:35.616412+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:35.616412+0000","last_undegraded":"2024-04-25T21:40:35.616412+0000","last_fullsized":"2024-04-25T21:40:35.616412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614816+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:35.614816+0000","last_peered":"2024-04-25T21:40:35.614816+0000","last_clean":"2024-04-25T21:40:35.614816+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:35.614816+0000","last_undegraded":"2024-04-25T21:40:35.614816+0000","last_fullsized":"2024-04-25T21:40:35.614816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.617201+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:35.617201+0000","last_peered":"2024-04-25T21:40:35.617201+0000","last_clean":"2024-04-25T21:40:35.617201+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:35.617201+0000","last_undegraded":"2024-04-25T21:40:35.617201+0000","last_fullsized":"2024-04-25T21:40:35.617201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:34.534127+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:34.534127+0000","last_peered":"2024-04-25T21:40:34.534127+0000","last_clean":"2024-04-25T21:40:34.534127+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:34.534127+0000","last_undegraded":"2024-04-25T21:40:34.534127+0000","last_fullsized":"2024-04-25T21:40:34.534127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613555+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:35.613555+0000","last_peered":"2024-04-25T21:40:35.613555+0000","last_clean":"2024-04-25T21:40:35.613555+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:35.613555+0000","last_undegraded":"2024-04-25T21:40:35.613555+0000","last_fullsized":"2024-04-25T21:40:35.613555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614625+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:35.614625+0000","last_peered":"2024-04-25T21:40:35.614625+0000","last_clean":"2024-04-25T21:40:35.614625+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:35.614625+0000","last_undegraded":"2024-04-25T21:40:35.614625+0000","last_fullsized":"2024-04-25T21:40:35.614625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613838+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:35.613838+0000","last_peered":"2024-04-25T21:40:35.613838+0000","last_clean":"2024-04-25T21:40:35.613838+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:35.613838+0000","last_undegraded":"2024-04-25T21:40:35.613838+0000","last_fullsized":"2024-04-25T21:40:35.613838+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:52.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T21:40:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.943+0000 7f77dfa6d640 10 monclient: get_monmap_and_config 2024-04-25T21:40:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.943+0000 7f77dfa6d640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.943+0000 7f77dfa6d640 10 monclient: monmap: 2024-04-25T21:40:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:52.946602+0000 2024-04-25T21:40:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:52.946602+0000 2024-04-25T21:40:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:52.946 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.943+0000 7f77dfa6d640 1 Processor -- start 2024-04-25T21:40:52.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.943+0000 7f77dfa6d640 1 -- start start 2024-04-25T21:40:52.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.943+0000 7f77dfa6d640 10 monclient: init 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient(hunting): picked mon.noname-c con 0x7f77d80a3e80 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient(hunting): picked mon.noname-b con 0x7f77d80a4970 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient(hunting): picked mon.noname-a con 0x7f77d813d680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d809adf0 con 0x7f77d813d680 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d809b340 con 0x7f77d80a3e80 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d809b620 con 0x7f77d80a4970 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient(hunting): authenticate will time out at 1011.856703s 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dd008640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f77d80a4970 0x7f77d813bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40864/0 (socket says 172.21.15.6:40864) 2024-04-25T21:40:52.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dd008640 1 -- 172.21.15.6:0/2411610614 learned_addr learned my addr 172.21.15.6:0/2411610614 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dd008640 1 -- 172.21.15.6:0/2411610614 >> v1:172.21.15.6:6790/0 conn(0x7f77d80a4970 legacy=0x7f77d813bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77de00a640 1 -- 172.21.15.6:0/2411610614 >> v1:172.21.15.6:6789/0 conn(0x7f77d813d680 legacy=0x7f77d813fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807217331 0 0) 0x7f77d809b620 con 0x7f77d80a4970 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): my global_id is 4514 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77c0003530 con 0x7f77d80a4970 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933570163 0 0) 0x7f77d809adf0 con 0x7f77d813d680 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient(hunting): my global_id is 4554 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77d809b620 con 0x7f77d813d680 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dd809640 1 -- 172.21.15.6:0/2411610614 >> v1:172.21.15.203:6789/0 conn(0x7f77d80a3e80 legacy=0x7f77d80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2178189409 0 0) 0x7f77c0003530 con 0x7f77d80a4970 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 >> v1:172.21.15.203:6789/0 conn(0x7f77d80a3e80 legacy=0x7f77d80a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 >> v1:172.21.15.6:6789/0 conn(0x7f77d813d680 legacy=0x7f77d813fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 monclient: found mon.noname-b 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d806eaf0 con 0x7f77d80a4970 2024-04-25T21:40:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f77cc0018d0 con 0x7f77d80a4970 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: dump: 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 5 monclient: authenticate success, global_id 4514 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77cc001c20 con 0x7f77d80a4970 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 1 -- 172.21.15.6:0/2411610614 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f77cc004a50 con 0x7f77d80a4970 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dc807640 10 monclient: dump: 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.947+0000 7f77dfa6d640 4 set_mon_vals no callback set 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient: shutdown 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- 172.21.15.6:0/2411610614 >> v1:172.21.15.6:6790/0 conn(0x7f77d80a4970 legacy=0x7f77d813bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- 172.21.15.6:0/2411610614 shutdown_connections 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- 172.21.15.6:0/2411610614 >> 172.21.15.6:0/2411610614 conn(0x7f77d80a5f80 msgr2=0x7f77d80a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- 172.21.15.6:0/2411610614 shutdown_connections 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- 172.21.15.6:0/2411610614 wait complete. 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient: build_initial_monmap 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient: monmap: 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:52.956 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 Processor -- start 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- start start 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient: init 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient(hunting): picked mon.noname-a con 0x7f77d80a3e80 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient(hunting): picked mon.noname-c con 0x7f77d813d680 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient(hunting): picked mon.noname-b con 0x7f77d8121670 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d8139310 con 0x7f77d80a3e80 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d8123d00 con 0x7f77d8121670 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d8139d00 con 0x7f77d813d680 2024-04-25T21:40:52.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dfa6d640 10 monclient(hunting): authenticate will time out at 1011.861990s 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dd809640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f77d80a3e80 0x7f77d8138c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55608/0 (socket says 172.21.15.6:55608) 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.951+0000 7f77dd809640 1 -- 172.21.15.6:0/1527278890 learned_addr learned my addr 172.21.15.6:0/1527278890 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dd809640 1 -- 172.21.15.6:0/1527278890 >> v1:172.21.15.6:6789/0 conn(0x7f77d80a3e80 legacy=0x7f77d8138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dd008640 1 -- 172.21.15.6:0/1527278890 >> v1:172.21.15.6:6790/0 conn(0x7f77d813d680 legacy=0x7f77d81d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546516007 0 0) 0x7f77d8139d00 con 0x7f77d813d680 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): my global_id is 4517 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77c4003530 con 0x7f77d813d680 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77de00a640 1 -- 172.21.15.6:0/1527278890 >> v1:172.21.15.203:6789/0 conn(0x7f77d8121670 legacy=0x7f77d81d4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3379876617 0 0) 0x7f77c4003530 con 0x7f77d813d680 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f77d8139d00 con 0x7f77d813d680 2024-04-25T21:40:52.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f77cc004990 con 0x7f77d813d680 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): dump: 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3787765720 0 0) 0x7f77d8139d00 con 0x7f77d813d680 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 >> v1:172.21.15.203:6789/0 conn(0x7f77d8121670 legacy=0x7f77d81d4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 >> v1:172.21.15.6:6789/0 conn(0x7f77d80a3e80 legacy=0x7f77d8138c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 monclient: found mon.c 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d80a5c00 con 0x7f77d813d680 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 5 monclient: authenticate success, global_id 4517 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 10 monclient: _renew_subs 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f77d81222b0 con 0x7f77d813d680 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 10 monclient: _renew_subs 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f77d81cc6f0 con 0x7f77d813d680 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f77cc001c00 con 0x7f77d813d680 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f77cc005220 con 0x7f77d813d680 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 10 monclient: dump: 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:52.964 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d77fe640 4 set_mon_vals no callback set 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b8005090 con 0x7f77d813d680 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f77cc00ffc0 con 0x7f77d813d680 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f77cc044370 con 0x7f77d813d680 2024-04-25T21:40:52.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.955+0000 7f77dd809640 10 monclient: get_auth_request con 0x7f77c4036a20 auth_method 0 2024-04-25T21:40:52.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.959+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f77cc015810 con 0x7f77d813d680 2024-04-25T21:40:52.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.959+0000 7f77d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:52.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:52.959+0000 7f77d67fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.119+0000 7f77dfa6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.119+0000 7f77dfa6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.119+0000 7f77dfa6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:53.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.119+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77b8005380 con 0x7f77d813d680 2024-04-25T21:40:53.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77d67fc640 1 -- 172.21.15.6:0/1527278890 <== mon.2 v1:172.21.15.6:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+10775 (unknown 780366164 0 350410764) 0x7f77cc015110 con 0x7f77d813d680 2024-04-25T21:40:53.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-25T21:40:53.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77d67fc640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:40:53.123 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:53.123 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":25,"fsid":"7c01789f-5916-4cc5-8455-2e5d8f8fad2b","created":"2024-04-25T21:39:46.151487+0000","modified":"2024-04-25T21:40:34.128568+0000","last_up_change":"2024-04-25T21:39:58.313547+0000","last_in_change":"2024-04-25T21:39:50.116466+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T21:39:59.601307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T21:40:05.399371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2b3e085e-25b5-4eff-bed1-787528f6c0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6804","nonce":763650169}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6805","nonce":763650169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6809","nonce":763650169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6807","nonce":763650169}]},"public_addr":"172.21.15.6:6804/763650169","cluster_addr":"172.21.15.6:6805/763650169","heartbeat_back_addr":"172.21.15.6:6809/763650169","heartbeat_front_addr":"172.21.15.6:6807/763650169","state":["exists","up"]},{"osd":1,"uuid":"6e5949dd-a414-4826-a94e-3e985f784fa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6806","nonce":2126016572}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6808","nonce":2126016572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6811","nonce":2126016572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6810","nonce":2126016572}]},"public_addr":"172.21.15.6:6806/2126016572","cluster_addr":"172.21.15.6:6808/2126016572","heartbeat_back_addr":"172.21.15.6:6811/2126016572","heartbeat_front_addr":"172.21.15.6:6810/2126016572","state":["exists","up"]},{"osd":2,"uuid":"44114590-b8c2-4e3a-8073-aecac03aa2a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6800","nonce":3408092622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6801","nonce":3408092622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6803","nonce":3408092622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.6:6802","nonce":3408092622}]},"public_addr":"172.21.15.6:6800/3408092622","cluster_addr":"172.21.15.6:6801/3408092622","heartbeat_back_addr":"172.21.15.6:6803/3408092622","heartbeat_front_addr":"172.21.15.6:6802/3408092622","state":["exists","up"]},{"osd":3,"uuid":"752f9b53-7f3b-4d16-8e45-84e24469a515","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":3201437079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":3201437079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":3201437079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":3201437079}]},"public_addr":"172.21.15.203:6800/3201437079","cluster_addr":"172.21.15.203:6802/3201437079","heartbeat_back_addr":"172.21.15.203:6806/3201437079","heartbeat_front_addr":"172.21.15.203:6804/3201437079","state":["exists","up"]},{"osd":4,"uuid":"63bc5f2b-57c1-432c-bd64-3c3753ceb48f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":1405305648}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":1405305648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":1405305648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":1405305648}]},"public_addr":"172.21.15.203:6801/1405305648","cluster_addr":"172.21.15.203:6803/1405305648","heartbeat_back_addr":"172.21.15.203:6807/1405305648","heartbeat_front_addr":"172.21.15.203:6805/1405305648","state":["exists","up"]},{"osd":5,"uuid":"77440232-26ad-4654-9cba-3444b17e5b1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":1687281622}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":1687281622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":1687281622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":1687281622}]},"public_addr":"172.21.15.203:6808/1687281622","cluster_addr":"172.21.15.203:6809/1687281622","heartbeat_back_addr":"172.21.15.203:6811/1687281622","heartbeat_front_addr":"172.21.15.203:6810/1687281622","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.481868+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.631515+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.371692+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.722323+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.712123+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T21:39:55.636554+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 >> v1:172.21.15.203:6812/3164820986 conn(0x7f77c4036a20 legacy=0x7f77c4038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 10 monclient: shutdown 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 >> v1:172.21.15.6:6790/0 conn(0x7f77d813d680 legacy=0x7f77d81d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 shutdown_connections 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 >> 172.21.15.6:0/1527278890 conn(0x7f77d80a5f80 msgr2=0x7f77d80a8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 shutdown_connections 2024-04-25T21:40:53.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:53.123+0000 7f77dfa6d640 1 -- 172.21.15.6:0/1527278890 wait complete. 2024-04-25T21:40:54.141 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T21:40:54.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T21:40:54.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient: get_monmap_and_config 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient: monmap: 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:54.284646+0000 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:54.284646+0000 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:54.318 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 1 Processor -- start 2024-04-25T21:40:54.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 1 -- start start 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient: init 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient(hunting): picked mon.noname-a con 0x7fe1100a15d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient(hunting): picked mon.noname-c con 0x7fe1100a5910 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient(hunting): picked mon.noname-b con 0x7fe1101338f0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe11009c5b0 con 0x7fe1100a15d0 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe11009d490 con 0x7fe1100a5910 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe11009d770 con 0x7fe1101338f0 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe11702d640 10 monclient(hunting): authenticate will time out at 1013.194578s 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe114dc9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe1100a15d0 0x7fe1100a2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55614/0 (socket says 172.21.15.6:55614) 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.283+0000 7fe114dc9640 1 -- 172.21.15.6:0/2970357704 learned_addr learned my addr 172.21.15.6:0/2970357704 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe114dc9640 1 -- 172.21.15.6:0/2970357704 >> v1:172.21.15.6:6789/0 conn(0x7fe1100a15d0 legacy=0x7fe1100a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe1155ca640 1 -- 172.21.15.6:0/2970357704 >> v1:172.21.15.6:6790/0 conn(0x7fe1101338f0 legacy=0x7fe11013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471417179 0 0) 0x7fe11009d770 con 0x7fe1101338f0 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): my global_id is 4520 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0fc005740 con 0x7fe1101338f0 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2303478929 0 0) 0x7fe11009c5b0 con 0x7fe1100a15d0 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient(hunting): my global_id is 4557 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe11009d770 con 0x7fe1100a15d0 2024-04-25T21:40:54.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10ffff640 1 -- 172.21.15.6:0/2970357704 >> v1:172.21.15.203:6789/0 conn(0x7fe1100a5910 legacy=0x7fe1101331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4092191094 0 0) 0x7fe0fc005740 con 0x7fe1101338f0 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 >> v1:172.21.15.203:6789/0 conn(0x7fe1100a5910 legacy=0x7fe1101331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 >> v1:172.21.15.6:6789/0 conn(0x7fe1100a15d0 legacy=0x7fe1100a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 monclient: found mon.noname-b 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1101408a0 con 0x7fe1101338f0 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe108001920 con 0x7fe1101338f0 2024-04-25T21:40:54.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 5 monclient: authenticate success, global_id 4520 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: dump: 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe108001c70 con 0x7fe1101338f0 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 1 -- 172.21.15.6:0/2970357704 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe1080037c0 con 0x7fe1101338f0 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe10f7fe640 10 monclient: dump: 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 4 set_mon_vals no callback set 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 10 monclient: shutdown 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 1 -- 172.21.15.6:0/2970357704 >> v1:172.21.15.6:6790/0 conn(0x7fe1101338f0 legacy=0x7fe11013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 1 -- 172.21.15.6:0/2970357704 shutdown_connections 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 1 -- 172.21.15.6:0/2970357704 >> 172.21.15.6:0/2970357704 conn(0x7fe1100a5500 msgr2=0x7fe1100a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 1 -- 172.21.15.6:0/2970357704 shutdown_connections 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 1 -- 172.21.15.6:0/2970357704 wait complete. 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 10 monclient: monmap: 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.331 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 1 Processor -- start 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 1 -- start start 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.287+0000 7fe11702d640 10 monclient: init 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient(hunting): picked mon.noname-a con 0x7fe1100a15d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient(hunting): picked mon.noname-b con 0x7fe1100a5910 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient(hunting): picked mon.noname-c con 0x7fe1101338f0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1101363d0 con 0x7fe1100a15d0 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe11009eb70 con 0x7fe1100a5910 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe11009f060 con 0x7fe1101338f0 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient(hunting): authenticate will time out at 1013.200503s 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe114dc9640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe1100a15d0 0x7fe11013fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55620/0 (socket says 172.21.15.6:55620) 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe1155ca640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fe1101338f0 0x7fe1101d04b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40874/0 (socket says 172.21.15.6:40874) 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe114dc9640 1 -- 172.21.15.6:0/1452089538 learned_addr learned my addr 172.21.15.6:0/1452089538 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe114dc9640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.6:6789/0 conn(0x7fe1100a15d0 legacy=0x7fe11013fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe1155ca640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.6:6790/0 conn(0x7fe1101338f0 legacy=0x7fe1101d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3476938757 0 0) 0x7fe1101363d0 con 0x7fe1100a15d0 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): my global_id is 4560 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0f0003530 con 0x7fe1100a15d0 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 768333975 0 0) 0x7fe11009f060 con 0x7fe1101338f0 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): my global_id is 4523 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1101363d0 con 0x7fe1101338f0 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10ffff640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.203:6789/0 conn(0x7fe1100a5910 legacy=0x7fe1101ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 499521790 0 0) 0x7fe0f0003530 con 0x7fe1100a15d0 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe11009f060 con 0x7fe1100a15d0 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe100001f10 con 0x7fe1100a15d0 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:54.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3792701551 0 0) 0x7fe1101363d0 con 0x7fe1101338f0 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe0f0003530 con 0x7fe1101338f0 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe1080037c0 con 0x7fe1101338f0 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:54.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): dump: 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4243038093 0 0) 0x7fe11009eb70 con 0x7fe1100a5910 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient(hunting): my global_id is 94159 2024-04-25T21:40:54.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1101363d0 con 0x7fe1100a5910 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2057913899 0 0) 0x7fe11009f060 con 0x7fe1100a15d0 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.6:6790/0 conn(0x7fe1101338f0 legacy=0x7fe1101d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.203:6789/0 conn(0x7fe1100a5910 legacy=0x7fe1101ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 monclient: found mon.a 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1101c8140 con 0x7fe1100a15d0 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 5 monclient: authenticate success, global_id 4560 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient: _renew_subs 2024-04-25T21:40:54.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1100721d0 con 0x7fe1100a15d0 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient: _renew_subs 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1101402c0 con 0x7fe1100a15d0 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe100004ea0 con 0x7fe1100a15d0 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe1000051f0 con 0x7fe1100a15d0 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10effd640 4 set_mon_vals no callback set 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.291+0000 7fe10d7fa640 10 monclient: dump: 2024-04-25T21:40:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe11702d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe0f4000f90 con 0x7fe1100a15d0 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fe100005410 con 0x7fe1100a15d0 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fe1000446f0 con 0x7fe1100a15d0 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6804/763650169 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0f0041420 con 0x7fe0f003d7b0 2024-04-25T21:40:54.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (unknown 2663167439 0 0) 0x7fe100044b20 con 0x7fe1100a15d0 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe10d7fa640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe1155ca640 10 monclient: get_auth_request con 0x7fe0f003d7b0 auth_method 0 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.295+0000 7fe10ffff640 10 monclient: get_auth_request con 0x7fe0f0036bd0 auth_method 0 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.299+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== osd.0 v1:172.21.15.6:6804/763650169 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe0f0041420 con 0x7fe0f003d7b0 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.311+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 --> v1:172.21.15.6:6804/763650169 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe0f4002cc0 con 0x7fe0f003d7b0 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe10d7fa640 1 -- 172.21.15.6:0/1452089538 <== osd.0 v1:172.21.15.6:6804/763650169 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fe0f4002cc0 con 0x7fe0f003d7b0 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T21:40:54.340 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.6:6804/763650169 conn(0x7fe0f003d7b0 legacy=0x7fe0f003fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.203:6812/3164820986 conn(0x7fe0f0036bd0 legacy=0x7fe0f0039090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 10 monclient: shutdown 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 >> v1:172.21.15.6:6789/0 conn(0x7fe1100a15d0 legacy=0x7fe11013fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe1155ca640 1 -- 172.21.15.6:0/1452089538 reap_dead start 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 shutdown_connections 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 >> 172.21.15.6:0/1452089538 conn(0x7fe1100a5500 msgr2=0x7fe11012f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 shutdown_connections 2024-04-25T21:40:54.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.315+0000 7fe11702d640 1 -- 172.21.15.6:0/1452089538 wait complete. 2024-04-25T21:40:54.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T21:40:54.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient: get_monmap_and_config 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient: monmap: 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:54.459783+0000 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:54.459783+0000 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:54.459 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:54.460 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 1 Processor -- start 2024-04-25T21:40:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 1 -- start start 2024-04-25T21:40:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient: init 2024-04-25T21:40:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient(hunting): picked mon.noname-a con 0x7f704006e890 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient(hunting): picked mon.noname-c con 0x7f704009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient(hunting): picked mon.noname-b con 0x7f704013d0a0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704009b0d0 con 0x7f704006e890 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70400a4ca0 con 0x7f704009bab0 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704006f070 con 0x7f704013d0a0 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f70482ca640 10 monclient(hunting): authenticate will time out at 1013.369716s 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f7046066640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f704006e890 0x7f70400a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55630/0 (socket says 172.21.15.6:55630) 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f7046066640 1 -- 172.21.15.6:0/3850915549 learned_addr learned my addr 172.21.15.6:0/3850915549 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.459+0000 7f7046867640 1 --1- 172.21.15.6:0/3850915549 >> v1:172.21.15.6:6790/0 conn(0x7f704013d0a0 0x7f704013f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40880/0 (socket says 172.21.15.6:40880) 2024-04-25T21:40:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7046867640 1 -- 172.21.15.6:0/3850915549 >> v1:172.21.15.6:6790/0 conn(0x7f704013d0a0 legacy=0x7f704013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7046066640 1 -- 172.21.15.6:0/3850915549 >> v1:172.21.15.6:6789/0 conn(0x7f704006e890 legacy=0x7f70400a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121202320 0 0) 0x7f703c001460 con 0x7f704006e890 2024-04-25T21:40:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): my global_id is 4563 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f702c003530 con 0x7f704006e890 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3201497884 0 0) 0x7f704006f070 con 0x7f704013d0a0 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient(hunting): my global_id is 4526 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045865640 1 -- 172.21.15.6:0/3850915549 >> v1:172.21.15.203:6789/0 conn(0x7f704009bab0 legacy=0x7f70400a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f703c001460 con 0x7f704013d0a0 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2062060544 0 0) 0x7f702c003530 con 0x7f704006e890 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 >> v1:172.21.15.6:6790/0 conn(0x7f704013d0a0 legacy=0x7f704013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 >> v1:172.21.15.203:6789/0 conn(0x7f704009bab0 legacy=0x7f70400a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 monclient: found mon.noname-a 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f704009adf0 con 0x7f704006e890 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 5 monclient: authenticate success, global_id 4563 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f703c001aa0 con 0x7f704006e890 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: dump: 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f703c0048b0 con 0x7f704006e890 2024-04-25T21:40:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 1 -- 172.21.15.6:0/3850915549 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f703c004c00 con 0x7f704006e890 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7045064640 10 monclient: dump: 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.469 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 4 set_mon_vals no callback set 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient: shutdown 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- 172.21.15.6:0/3850915549 >> v1:172.21.15.6:6789/0 conn(0x7f704006e890 legacy=0x7f70400a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- 172.21.15.6:0/3850915549 shutdown_connections 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- 172.21.15.6:0/3850915549 >> 172.21.15.6:0/3850915549 conn(0x7f70400a5f80 msgr2=0x7f70400a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- 172.21.15.6:0/3850915549 shutdown_connections 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- 172.21.15.6:0/3850915549 wait complete. 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient: monmap: 2024-04-25T21:40:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 Processor -- start 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- start start 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient: init 2024-04-25T21:40:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient(hunting): picked mon.noname-c con 0x7f704006e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient(hunting): picked mon.noname-b con 0x7f704009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient(hunting): picked mon.noname-a con 0x7f704013d0a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704013ed50 con 0x7f704013d0a0 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704013e080 con 0x7f704009bab0 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704013e360 con 0x7f704006e890 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f70482ca640 10 monclient(hunting): authenticate will time out at 1013.375005s 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7046867640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f704013d0a0 0x7f70401d3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55646/0 (socket says 172.21.15.6:55646) 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7046867640 1 -- 172.21.15.6:0/3943531004 learned_addr learned my addr 172.21.15.6:0/3943531004 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.463+0000 7f7046066640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f704006e890 0x7f70400a0a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40894/0 (socket says 172.21.15.6:40894) 2024-04-25T21:40:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7046867640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f704013d0a0 legacy=0x7f70401d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7046066640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6790/0 conn(0x7f704006e890 legacy=0x7f70400a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726867713 0 0) 0x7f704013ed50 con 0x7f704013d0a0 2024-04-25T21:40:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): my global_id is 4566 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7024003530 con 0x7f704013d0a0 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 215610685 0 0) 0x7f704013e360 con 0x7f704006e890 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): my global_id is 4529 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f704013ed50 con 0x7f704006e890 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7045865640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.203:6789/0 conn(0x7f704009bab0 legacy=0x7f70400a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 521786072 0 0) 0x7f7024003530 con 0x7f704013d0a0 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f704013e360 con 0x7f704013d0a0 2024-04-25T21:40:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7038001fc0 con 0x7f704013d0a0 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.479 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2558238690 0 0) 0x7f704013ed50 con 0x7f704006e890 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7024003530 con 0x7f704006e890 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f703c0014d0 con 0x7f704006e890 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): dump: 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3393584890 0 0) 0x7f704013e360 con 0x7f704013d0a0 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6790/0 conn(0x7f704006e890 legacy=0x7f70400a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.203:6789/0 conn(0x7f704009bab0 legacy=0x7f70400a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 monclient: found mon.a 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70400a2df0 con 0x7f704013d0a0 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 5 monclient: authenticate success, global_id 4566 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 10 monclient: _renew_subs 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f704009c490 con 0x7f704013d0a0 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 10 monclient: _renew_subs 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f70400a1870 con 0x7f704013d0a0 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7038004e20 con 0x7f704013d0a0 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7038005170 con 0x7f704013d0a0 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: dump: 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7044863640 4 set_mon_vals no callback set 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7046867640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f704013d0a0 legacy=0x7f70401d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7046867640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f704013d0a0 legacy=0x7f70401d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T21:40:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7046867640 1 --1- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f704013d0a0 0x7f70401d3c70 :-1 s=OPENED pgs=213 cs=1 l=1).handle_message read tag failed 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7046867640 1 --1- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f704013d0a0 0x7f70401d3c70 :-1 s=OPENED pgs=213 cs=1 l=1).fault on lossy channel, failing 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.6:6789/0 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f704013d0a0 legacy=0x7f70401d3c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): picked mon.a con 0x7f7024003ae0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): picked mon.b con 0x7f704009bab0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): picked mon.c con 0x7f704006e890 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f704013e360 con 0x7f7024003ae0 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f704013e080 con 0x7f704009bab0 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7024003530 con 0x7f704006e890 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f7036ffd640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.467+0000 7f70482ca640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:54.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7046066640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f7024003ae0 legacy=0x7f7024003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7046867640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6790/0 conn(0x7f704006e890 legacy=0x7f702400e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774536452 0 0) 0x7f704013e360 con 0x7f7024003ae0 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f7024012290 con 0x7f7024003ae0 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7045865640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.203:6789/0 conn(0x7f704009bab0 legacy=0x7f702400aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051711787 0 0) 0x7f7024003530 con 0x7f704006e890 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f704013e360 con 0x7f704006e890 2024-04-25T21:40:54.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1287434776 0 0) 0x7f7024012290 con 0x7f7024003ae0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6790/0 conn(0x7f704006e890 legacy=0x7f702400e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.203:6789/0 conn(0x7f704009bab0 legacy=0x7f702400aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 monclient: found mon.a 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f704013ed50 con 0x7f7024003ae0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7028005090 con 0x7f7024003ae0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7024003530 con 0x7f7024003ae0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f703c0050e0 con 0x7f7024003ae0 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: dump: 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f703c0050e0 con 0x7f7024003ae0 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7037fff640 4 set_mon_vals no callback set 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f703c00fe90 con 0x7f7024003ae0 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 6 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f703c044390 con 0x7f7024003ae0 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.471+0000 7f7045865640 10 monclient: get_auth_request con 0x7f704013d0a0 auth_method 0 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.475+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f704009b0d0 con 0x7f7024003ae0 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.475+0000 7f7036ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.475+0000 7f7036ffd640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.479+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mon.0 v1:172.21.15.6:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f703c015830 con 0x7f7024003ae0 2024-04-25T21:40:54.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.479+0000 7f7036ffd640 10 monclient: handle_mon_command_ack 1 not found 2024-04-25T21:40:54.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.635+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f7028000d60 con 0x7f704013d0a0 2024-04-25T21:40:54.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.635+0000 7f7036ffd640 1 -- 172.21.15.6:0/3943531004 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f7028000d60 con 0x7f704013d0a0 2024-04-25T21:40:54.637 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T21:40:54.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.203:6812/3164820986 conn(0x7f704013d0a0 legacy=0x7f70240442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 10 monclient: shutdown 2024-04-25T21:40:54.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:54.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 >> v1:172.21.15.6:6789/0 conn(0x7f7024003ae0 legacy=0x7f7024003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 shutdown_connections 2024-04-25T21:40:54.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 >> 172.21.15.6:0/3943531004 conn(0x7f70400a5f80 msgr2=0x7f7040136360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:54.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 shutdown_connections 2024-04-25T21:40:54.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.639+0000 7f70482ca640 1 -- 172.21.15.6:0/3943531004 wait complete. 2024-04-25T21:40:54.654 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T21:40:54.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T21:40:54.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient: get_monmap_and_config 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient: monmap: 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:54.776063+0000 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:54.776063+0000 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:54.775 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 1 Processor -- start 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 1 -- start start 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient: init 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f5fbc09c400 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f5fbc09c7e0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f5fbc139c10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fbc0a0a10 con 0x7f5fbc139c10 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fbc09d810 con 0x7f5fbc09c7e0 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fbc09daf0 con 0x7f5fbc09c400 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc3ef6640 10 monclient(hunting): authenticate will time out at 1013.685561s 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc1c92640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5fbc09c400 0x7f5fbc09ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40910/0 (socket says 172.21.15.6:40910) 2024-04-25T21:40:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc1c92640 1 -- 172.21.15.6:0/3965896821 learned_addr learned my addr 172.21.15.6:0/3965896821 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.775+0000 7f5fc1c92640 1 -- 172.21.15.6:0/3965896821 >> v1:172.21.15.6:6790/0 conn(0x7f5fbc09c400 legacy=0x7f5fbc09ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc2493640 1 -- 172.21.15.6:0/3965896821 >> v1:172.21.15.6:6789/0 conn(0x7f5fbc139c10 legacy=0x7f5fbc13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731409252 0 0) 0x7f5fbc09daf0 con 0x7f5fbc09c400 2024-04-25T21:40:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): my global_id is 4532 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fa8003530 con 0x7f5fbc09c400 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692211113 0 0) 0x7f5fbc0a0a10 con 0x7f5fbc139c10 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient(hunting): my global_id is 4569 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fbc09daf0 con 0x7f5fbc139c10 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2404396650 0 0) 0x7f5fa8003530 con 0x7f5fbc09c400 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 >> v1:172.21.15.203:6789/0 conn(0x7f5fbc09c7e0 legacy=0x7f5fbc09cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc1491640 1 -- 172.21.15.6:0/3965896821 >> v1:172.21.15.203:6789/0 conn(0x7f5fbc09c7e0 legacy=0x7f5fbc09cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 >> v1:172.21.15.6:6789/0 conn(0x7f5fbc139c10 legacy=0x7f5fbc13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 monclient: found mon.noname-b 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc0a2640 con 0x7f5fbc09c400 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fb80018d0 con 0x7f5fbc09c400 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 5 monclient: authenticate success, global_id 4532 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: dump: 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.783 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fb8001be0 con 0x7f5fbc09c400 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 1 -- 172.21.15.6:0/3965896821 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fb8004a50 con 0x7f5fbc09c400 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc0c90640 10 monclient: dump: 2024-04-25T21:40:54.784 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 4 set_mon_vals no callback set 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient: shutdown 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/3965896821 >> v1:172.21.15.6:6790/0 conn(0x7f5fbc09c400 legacy=0x7f5fbc09ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/3965896821 shutdown_connections 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/3965896821 >> 172.21.15.6:0/3965896821 conn(0x7f5fbc0a5500 msgr2=0x7f5fbc0a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/3965896821 shutdown_connections 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/3965896821 wait complete. 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient: monmap: 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 Processor -- start 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- start start 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient: init 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f5fbc09c400 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f5fbc09c7e0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f5fbc139c10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fbc071890 con 0x7f5fbc09c400 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fbc071b70 con 0x7f5fbc139c10 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fbc06eda0 con 0x7f5fbc09c7e0 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc3ef6640 10 monclient(hunting): authenticate will time out at 1013.690633s 2024-04-25T21:40:54.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc1c92640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5fbc09c400 0x7f5fbc06df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55678/0 (socket says 172.21.15.6:55678) 2024-04-25T21:40:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc1491640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5fbc09c7e0 0x7f5fbc06e690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40912/0 (socket says 172.21.15.6:40912) 2024-04-25T21:40:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.779+0000 7f5fc1c92640 1 -- 172.21.15.6:0/2564880639 learned_addr learned my addr 172.21.15.6:0/2564880639 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc1491640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.6:6790/0 conn(0x7f5fbc09c7e0 legacy=0x7f5fbc06e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc1c92640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.6:6789/0 conn(0x7f5fbc09c400 legacy=0x7f5fbc06df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 410512451 0 0) 0x7f5fbc071890 con 0x7f5fbc09c400 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): my global_id is 4572 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fa0003560 con 0x7f5fbc09c400 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2790400375 0 0) 0x7f5fbc06eda0 con 0x7f5fbc09c7e0 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): my global_id is 4535 2024-04-25T21:40:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fbc071890 con 0x7f5fbc09c7e0 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc2493640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.203:6789/0 conn(0x7f5fbc139c10 legacy=0x7f5fbc1cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2348249618 0 0) 0x7f5fa0003560 con 0x7f5fbc09c400 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fbc06eda0 con 0x7f5fbc09c400 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fb8004990 con 0x7f5fbc09c400 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.792 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2775235203 0 0) 0x7f5fbc071890 con 0x7f5fbc09c7e0 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fa0003560 con 0x7f5fbc09c7e0 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fac001aa0 con 0x7f5fbc09c7e0 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 123632886 0 0) 0x7f5fbc06eda0 con 0x7f5fbc09c400 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.6:6790/0 conn(0x7f5fbc09c7e0 legacy=0x7f5fbc06e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.203:6789/0 conn(0x7f5fbc139c10 legacy=0x7f5fbc1cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 monclient: found mon.a 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc070650 con 0x7f5fbc09c400 2024-04-25T21:40:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 5 monclient: authenticate success, global_id 4572 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 10 monclient: _renew_subs 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5fbc06faf0 con 0x7f5fbc09c400 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 10 monclient: _renew_subs 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5fbc071210 con 0x7f5fbc09c400 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fb8001be0 con 0x7f5fbc09c400 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5fb80051e0 con 0x7f5fbc09c400 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: dump: 2024-04-25T21:40:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb3fff640 4 set_mon_vals no callback set 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5fa4000f90 con 0x7f5fbc09c400 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f5fb800ff80 con 0x7f5fbc09c400 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f5fb8044480 con 0x7f5fbc09c400 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6806/2126016572 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fa0041400 con 0x7f5fa003d790 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (unknown 2663167439 0 0) 0x7f5fb80448b0 con 0x7f5fbc09c400 2024-04-25T21:40:54.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fb27fc640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-04-25T21:40:54.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc2493640 10 monclient: get_auth_request con 0x7f5fa003d790 auth_method 0 2024-04-25T21:40:54.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.783+0000 7f5fc1491640 10 monclient: get_auth_request con 0x7f5fa0036b40 auth_method 0 2024-04-25T21:40:54.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.787+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== osd.1 v1:172.21.15.6:6806/2126016572 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5fa0041400 con 0x7f5fa003d790 2024-04-25T21:40:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 --> v1:172.21.15.6:6806/2126016572 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5fa4002cc0 con 0x7f5fa003d790 2024-04-25T21:40:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fb27fc640 1 -- 172.21.15.6:0/2564880639 <== osd.1 v1:172.21.15.6:6806/2126016572 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f5fa4002cc0 con 0x7f5fa003d790 2024-04-25T21:40:54.803 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:54.803 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T21:40:54.803 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.6:6806/2126016572 conn(0x7f5fa003d790 legacy=0x7f5fa003fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.203:6812/3164820986 conn(0x7f5fa0036b40 legacy=0x7f5fa0039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 10 monclient: shutdown 2024-04-25T21:40:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 >> v1:172.21.15.6:6789/0 conn(0x7f5fbc09c400 legacy=0x7f5fbc06df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc2493640 1 -- 172.21.15.6:0/2564880639 reap_dead start 2024-04-25T21:40:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 shutdown_connections 2024-04-25T21:40:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 >> 172.21.15.6:0/2564880639 conn(0x7f5fbc0a5500 msgr2=0x7f5fbc133a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 shutdown_connections 2024-04-25T21:40:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.803+0000 7f5fc3ef6640 1 -- 172.21.15.6:0/2564880639 wait complete. 2024-04-25T21:40:54.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T21:40:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.923+0000 7f5f35384640 10 monclient: get_monmap_and_config 2024-04-25T21:40:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.923+0000 7f5f35384640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.923+0000 7f5f35384640 10 monclient: monmap: 2024-04-25T21:40:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:54.927433+0000 2024-04-25T21:40:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:54.927433+0000 2024-04-25T21:40:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:54.927 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 1 Processor -- start 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 1 -- start start 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient: init 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient(hunting): picked mon.noname-b con 0x7f5f3009e860 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient(hunting): picked mon.noname-c con 0x7f5f300a1270 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient(hunting): picked mon.noname-a con 0x7f5f300a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f3009c130 con 0x7f5f300a1680 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f30099520 con 0x7f5f300a1270 2024-04-25T21:40:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f30099b10 con 0x7f5f3009e860 2024-04-25T21:40:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 10 monclient(hunting): authenticate will time out at 1013.836924s 2024-04-25T21:40:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2f7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5f300a1680 0x7f5f3013d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55682/0 (socket says 172.21.15.6:55682) 2024-04-25T21:40:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5f3009e860 0x7f5f300a0b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40922/0 (socket says 172.21.15.6:40922) 2024-04-25T21:40:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2f7fe640 1 -- 172.21.15.6:0/1519666396 learned_addr learned my addr 172.21.15.6:0/1519666396 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2f7fe640 1 -- 172.21.15.6:0/1519666396 >> v1:172.21.15.6:6789/0 conn(0x7f5f300a1680 legacy=0x7f5f3013d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2effd640 1 -- 172.21.15.6:0/1519666396 >> v1:172.21.15.6:6790/0 conn(0x7f5f3009e860 legacy=0x7f5f300a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788649009 0 0) 0x7f5f280017e0 con 0x7f5f300a1680 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): my global_id is 4575 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f18003530 con 0x7f5f300a1680 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1247218829 0 0) 0x7f5f30099b10 con 0x7f5f3009e860 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient(hunting): my global_id is 4538 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f280017e0 con 0x7f5f3009e860 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2e7fc640 1 -- 172.21.15.6:0/1519666396 >> v1:172.21.15.203:6789/0 conn(0x7f5f300a1270 legacy=0x7f5f30139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1185521678 0 0) 0x7f5f18003530 con 0x7f5f300a1680 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 >> v1:172.21.15.6:6790/0 conn(0x7f5f3009e860 legacy=0x7f5f300a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 >> v1:172.21.15.203:6789/0 conn(0x7f5f300a1270 legacy=0x7f5f30139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 monclient: found mon.noname-a 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3009a0d0 con 0x7f5f300a1680 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5f28001e20 con 0x7f5f300a1680 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 5 monclient: authenticate success, global_id 4575 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f35384640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.927+0000 7f5f2dffb640 10 monclient: dump: 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.935 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f28004c30 con 0x7f5f300a1680 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2dffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2dffb640 1 -- 172.21.15.6:0/1519666396 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5f28004f80 con 0x7f5f300a1680 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 4 set_mon_vals no callback set 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2dffb640 10 monclient: dump: 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient: shutdown 2024-04-25T21:40:54.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- 172.21.15.6:0/1519666396 >> v1:172.21.15.6:6789/0 conn(0x7f5f300a1680 legacy=0x7f5f3013d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- 172.21.15.6:0/1519666396 shutdown_connections 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- 172.21.15.6:0/1519666396 >> 172.21.15.6:0/1519666396 conn(0x7f5f300a1c50 msgr2=0x7f5f300a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- 172.21.15.6:0/1519666396 shutdown_connections 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- 172.21.15.6:0/1519666396 wait complete. 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient: build_initial_monmap 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient: monmap: 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:54.938 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 Processor -- start 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- start start 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient: init 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient(hunting): picked mon.noname-c con 0x7f5f3009e860 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient(hunting): picked mon.noname-b con 0x7f5f300a1270 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient(hunting): picked mon.noname-a con 0x7f5f300a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f30095e90 con 0x7f5f300a1680 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f30096930 con 0x7f5f300a1270 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f30093c40 con 0x7f5f3009e860 2024-04-25T21:40:54.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:54.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f35384640 10 monclient(hunting): authenticate will time out at 1013.842008s 2024-04-25T21:40:54.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2effd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f5f3009e860 0x7f5f30092e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40926/0 (socket says 172.21.15.6:40926) 2024-04-25T21:40:54.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.931+0000 7f5f2effd640 1 -- 172.21.15.6:0/3101965241 learned_addr learned my addr 172.21.15.6:0/3101965241 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f2f7fe640 1 -- 172.21.15.6:0/3101965241 >> v1:172.21.15.6:6789/0 conn(0x7f5f300a1680 legacy=0x7f5f301d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f2effd640 1 -- 172.21.15.6:0/3101965241 >> v1:172.21.15.6:6790/0 conn(0x7f5f3009e860 legacy=0x7f5f30092e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 624518408 0 0) 0x7f5f30095e90 con 0x7f5f300a1680 2024-04-25T21:40:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): my global_id is 4578 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f1c003530 con 0x7f5f300a1680 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076642860 0 0) 0x7f5f30093c40 con 0x7f5f3009e860 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): my global_id is 4541 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f30095e90 con 0x7f5f3009e860 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f2e7fc640 1 -- 172.21.15.6:0/3101965241 >> v1:172.21.15.203:6789/0 conn(0x7f5f300a1270 legacy=0x7f5f30093530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2338661887 0 0) 0x7f5f1c003530 con 0x7f5f300a1680 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f30093c40 con 0x7f5f300a1680 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5f28001c40 con 0x7f5f300a1680 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): dump: 2024-04-25T21:40:54.944 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3776698237 0 0) 0x7f5f30095e90 con 0x7f5f3009e860 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f1c003530 con 0x7f5f3009e860 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5f24001c10 con 0x7f5f3009e860 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:54.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient(hunting): dump: 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2398961929 0 0) 0x7f5f30093c40 con 0x7f5f300a1680 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 >> v1:172.21.15.6:6790/0 conn(0x7f5f3009e860 legacy=0x7f5f30092e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 >> v1:172.21.15.203:6789/0 conn(0x7f5f300a1270 legacy=0x7f5f30093530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:54.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 monclient: found mon.a 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f30092170 con 0x7f5f300a1680 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 5 monclient: authenticate success, global_id 4578 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 10 monclient: _renew_subs 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f30072b40 con 0x7f5f300a1680 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 10 monclient: _renew_subs 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f300956e0 con 0x7f5f300a1680 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5f28004c30 con 0x7f5f300a1680 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f5f280056d0 con 0x7f5f300a1680 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f2d7fa640 4 set_mon_vals no callback set 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 10 monclient: dump: 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:54.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f5f28010470 con 0x7f5f300a1680 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f10005090 con 0x7f5f300a1680 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f5f28044730 con 0x7f5f300a1680 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.935+0000 7f5f2effd640 10 monclient: get_auth_request con 0x7f5f1c036b60 auth_method 0 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.939+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f5f3009c130 con 0x7f5f300a1680 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.939+0000 7f5f177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:54.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:54.939+0000 7f5f177fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5f10000ba0 con 0x7f5f1c036b60 2024-04-25T21:40:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f177fe640 1 -- 172.21.15.6:0/3101965241 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f5f10000ba0 con 0x7f5f1c036b60 2024-04-25T21:40:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T21:40:55.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 >> v1:172.21.15.203:6812/3164820986 conn(0x7f5f1c036b60 legacy=0x7f5f1c039020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 10 monclient: shutdown 2024-04-25T21:40:55.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 >> v1:172.21.15.6:6789/0 conn(0x7f5f300a1680 legacy=0x7f5f301d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 shutdown_connections 2024-04-25T21:40:55.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 >> 172.21.15.6:0/3101965241 conn(0x7f5f300a1c50 msgr2=0x7f5f30098810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 shutdown_connections 2024-04-25T21:40:55.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.099+0000 7f5f35384640 1 -- 172.21.15.6:0/3101965241 wait complete. 2024-04-25T21:40:55.116 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T21:40:55.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient: get_monmap_and_config 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient: monmap: 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:55.237291+0000 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:55.237291+0000 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.236 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:55.237 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:55.237 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 1 Processor -- start 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 1 -- start start 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient: init 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf80a3e10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient(hunting): picked mon.noname-b con 0x7f4cf80a5750 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient(hunting): picked mon.noname-c con 0x7f4cf809bb90 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf806eca0 con 0x7f4cf80a3e10 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf809adf0 con 0x7f4cf809bb90 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf809b0d0 con 0x7f4cf80a5750 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cfe1f7640 10 monclient(hunting): authenticate will time out at 1014.147191s 2024-04-25T21:40:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cf6ffd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4cf80a5750 0x7f4cf813c020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40930/0 (socket says 172.21.15.6:40930) 2024-04-25T21:40:55.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.235+0000 7f4cf6ffd640 1 -- 172.21.15.6:0/3491192086 learned_addr learned my addr 172.21.15.6:0/3491192086 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf77fe640 1 -- 172.21.15.6:0/3491192086 >> v1:172.21.15.6:6789/0 conn(0x7f4cf80a3e10 legacy=0x7f4cf80a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf6ffd640 1 -- 172.21.15.6:0/3491192086 >> v1:172.21.15.6:6790/0 conn(0x7f4cf80a5750 legacy=0x7f4cf813c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503598546 0 0) 0x7f4cf806eca0 con 0x7f4cf80a3e10 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): my global_id is 4581 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ce4003530 con 0x7f4cf80a3e10 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4012390700 0 0) 0x7f4cf809b0d0 con 0x7f4cf80a5750 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient(hunting): my global_id is 4544 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cf806eca0 con 0x7f4cf80a5750 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf7fff640 1 -- 172.21.15.6:0/3491192086 >> v1:172.21.15.203:6789/0 conn(0x7f4cf809bb90 legacy=0x7f4cf813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3265803815 0 0) 0x7f4ce4003530 con 0x7f4cf80a3e10 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 >> v1:172.21.15.6:6790/0 conn(0x7f4cf80a5750 legacy=0x7f4cf813c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 >> v1:172.21.15.203:6789/0 conn(0x7f4cf809bb90 legacy=0x7f4cf813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 monclient: found mon.noname-a 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf80a6880 con 0x7f4cf80a3e10 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ce8001920 con 0x7f4cf80a3e10 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 5 monclient: authenticate success, global_id 4581 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: dump: 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.244 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4ce8001c70 con 0x7f4cf80a3e10 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 4 set_mon_vals no callback set 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 1 -- 172.21.15.6:0/3491192086 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ce80037c0 con 0x7f4cf80a3e10 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cf67fc640 10 monclient: dump: 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.245 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 10 monclient: shutdown 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/3491192086 >> v1:172.21.15.6:6789/0 conn(0x7f4cf80a3e10 legacy=0x7f4cf80a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/3491192086 shutdown_connections 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/3491192086 >> 172.21.15.6:0/3491192086 conn(0x7f4cf80a9cc0 msgr2=0x7f4cf80a5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/3491192086 shutdown_connections 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/3491192086 wait complete. 2024-04-25T21:40:55.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 10 monclient: monmap: 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 1 Processor -- start 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 1 -- start start 2024-04-25T21:40:55.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.239+0000 7f4cfe1f7640 10 monclient: init 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient(hunting): picked mon.noname-c con 0x7f4cf809bb90 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient(hunting): picked mon.noname-b con 0x7f4cf80a3e10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient(hunting): picked mon.noname-a con 0x7f4cf80a5750 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf813d690 con 0x7f4cf80a5750 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf813df70 con 0x7f4cf80a3e10 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf813f0e0 con 0x7f4cf809bb90 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient(hunting): authenticate will time out at 1014.152417s 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cf77fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f4cf809bb90 0x7f4cf81cc410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40940/0 (socket says 172.21.15.6:40940) 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cf77fe640 1 -- 172.21.15.6:0/1863830759 learned_addr learned my addr 172.21.15.6:0/1863830759 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cf7fff640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.6:6789/0 conn(0x7f4cf80a5750 legacy=0x7f4cf81d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cf77fe640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.6:6790/0 conn(0x7f4cf809bb90 legacy=0x7f4cf81cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4270803043 0 0) 0x7f4cf813d690 con 0x7f4cf80a5750 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): my global_id is 4584 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cd8003530 con 0x7f4cf80a5750 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3277002155 0 0) 0x7f4cf813f0e0 con 0x7f4cf809bb90 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): my global_id is 4547 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cf813d690 con 0x7f4cf809bb90 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cf6ffd640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.203:6789/0 conn(0x7f4cf80a3e10 legacy=0x7f4cf81d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2576902743 0 0) 0x7f4cd8003530 con 0x7f4cf80a5750 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cf813f0e0 con 0x7f4cf80a5750 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4cf0001e80 con 0x7f4cf80a5750 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): dump: 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4224189612 0 0) 0x7f4cf813d690 con 0x7f4cf809bb90 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4cd8003530 con 0x7f4cf809bb90 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4ce80037c0 con 0x7f4cf809bb90 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): dump: 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 815986895 0 0) 0x7f4cf813df70 con 0x7f4cf80a3e10 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient(hunting): my global_id is 94180 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cf813d690 con 0x7f4cf80a3e10 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2323216682 0 0) 0x7f4cf813f0e0 con 0x7f4cf80a5750 2024-04-25T21:40:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.6:6790/0 conn(0x7f4cf809bb90 legacy=0x7f4cf81cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.203:6789/0 conn(0x7f4cf80a3e10 legacy=0x7f4cf81d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 monclient: found mon.a 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf813f3c0 con 0x7f4cf80a5750 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 5 monclient: authenticate success, global_id 4584 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient: _renew_subs 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf8137cd0 con 0x7f4cf80a5750 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient: _renew_subs 2024-04-25T21:40:55.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4cf81cccf0 con 0x7f4cf80a5750 2024-04-25T21:40:55.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4cf0003cd0 con 0x7f4cf80a5750 2024-04-25T21:40:55.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f4cf0004020 con 0x7f4cf80a5750 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4cf5ffb640 4 set_mon_vals no callback set 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.243+0000 7f4ceffff640 10 monclient: dump: 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4cfe1f7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4cdc000f90 con 0x7f4cf80a5750 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f4cf00041c0 con 0x7f4cf80a5750 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f4cf0044750 con 0x7f4cf80a5750 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6800/3408092622 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cd80415a0 con 0x7f4cd803d930 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (unknown 2663167439 0 0) 0x7f4cf0044b00 con 0x7f4cf80a5750 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4ceffff640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4cf6ffd640 10 monclient: get_auth_request con 0x7f4cd803d930 auth_method 0 2024-04-25T21:40:55.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.247+0000 7f4cf77fe640 10 monclient: get_auth_request con 0x7f4cd8036ce0 auth_method 0 2024-04-25T21:40:55.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.251+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== osd.2 v1:172.21.15.6:6800/3408092622 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4cd80415a0 con 0x7f4cd803d930 2024-04-25T21:40:55.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 --> v1:172.21.15.6:6800/3408092622 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4cdc002cc0 con 0x7f4cd803d930 2024-04-25T21:40:55.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4ceffff640 1 -- 172.21.15.6:0/1863830759 <== osd.2 v1:172.21.15.6:6800/3408092622 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f4cdc002cc0 con 0x7f4cd803d930 2024-04-25T21:40:55.265 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:55.265 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T21:40:55.265 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.6:6800/3408092622 conn(0x7f4cd803d930 legacy=0x7f4cd803fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.203:6812/3164820986 conn(0x7f4cd8036ce0 legacy=0x7f4cd80391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 10 monclient: shutdown 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 >> v1:172.21.15.6:6789/0 conn(0x7f4cf80a5750 legacy=0x7f4cf81d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cf7fff640 1 -- 172.21.15.6:0/1863830759 reap_dead start 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 shutdown_connections 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 >> 172.21.15.6:0/1863830759 conn(0x7f4cf80a9cc0 msgr2=0x7f4cf80a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 shutdown_connections 2024-04-25T21:40:55.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.263+0000 7f4cfe1f7640 1 -- 172.21.15.6:0/1863830759 wait complete. 2024-04-25T21:40:55.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient: get_monmap_and_config 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient: monmap: 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:55.392209+0000 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:55.392209+0000 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:55.392 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 1 Processor -- start 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 1 -- start start 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient: init 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient(hunting): picked mon.noname-a con 0x7f35a409b690 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient(hunting): picked mon.noname-b con 0x7f35a409ba70 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient(hunting): picked mon.noname-c con 0x7f35a409beb0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a4099bc0 con 0x7f35a409b690 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a40998e0 con 0x7f35a409beb0 2024-04-25T21:40:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a40a0dc0 con 0x7f35a409ba70 2024-04-25T21:40:55.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a9516640 10 monclient(hunting): authenticate will time out at 1014.302071s 2024-04-25T21:40:55.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a27fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f35a409ba70 0x7f35a413bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40942/0 (socket says 172.21.15.6:40942) 2024-04-25T21:40:55.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a2ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f35a409b690 0x7f35a4136e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55720/0 (socket says 172.21.15.6:55720) 2024-04-25T21:40:55.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.391+0000 7f35a27fc640 1 -- 172.21.15.6:0/2074959081 learned_addr learned my addr 172.21.15.6:0/2074959081 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a27fc640 1 -- 172.21.15.6:0/2074959081 >> v1:172.21.15.6:6790/0 conn(0x7f35a409ba70 legacy=0x7f35a413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a2ffd640 1 -- 172.21.15.6:0/2074959081 >> v1:172.21.15.6:6789/0 conn(0x7f35a409b690 legacy=0x7f35a4136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4168028511 0 0) 0x7f35a40a0dc0 con 0x7f35a409ba70 2024-04-25T21:40:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): my global_id is 4550 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3590003530 con 0x7f35a409ba70 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 375011799 0 0) 0x7f35a4099bc0 con 0x7f35a409b690 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient(hunting): my global_id is 4587 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35a40a0dc0 con 0x7f35a409b690 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a37fe640 1 -- 172.21.15.6:0/2074959081 >> v1:172.21.15.203:6789/0 conn(0x7f35a409beb0 legacy=0x7f35a413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4093630434 0 0) 0x7f3590003530 con 0x7f35a409ba70 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 >> v1:172.21.15.203:6789/0 conn(0x7f35a409beb0 legacy=0x7f35a413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 >> v1:172.21.15.6:6789/0 conn(0x7f35a409b690 legacy=0x7f35a4136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 monclient: found mon.noname-b 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a4143b20 con 0x7f35a409ba70 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f358c0018d0 con 0x7f35a409ba70 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 5 monclient: authenticate success, global_id 4550 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: dump: 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f358c001c20 con 0x7f35a409ba70 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 1 -- 172.21.15.6:0/2074959081 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f358c004a50 con 0x7f35a409ba70 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a1ffb640 10 monclient: dump: 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 4 set_mon_vals no callback set 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient: shutdown 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- 172.21.15.6:0/2074959081 >> v1:172.21.15.6:6790/0 conn(0x7f35a409ba70 legacy=0x7f35a413bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- 172.21.15.6:0/2074959081 shutdown_connections 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- 172.21.15.6:0/2074959081 >> 172.21.15.6:0/2074959081 conn(0x7f35a40a1c50 msgr2=0x7f35a40a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- 172.21.15.6:0/2074959081 shutdown_connections 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- 172.21.15.6:0/2074959081 wait complete. 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient: monmap: 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:55.402 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 Processor -- start 2024-04-25T21:40:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- start start 2024-04-25T21:40:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient: init 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient(hunting): picked mon.noname-a con 0x7f35a409b690 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient(hunting): picked mon.noname-b con 0x7f35a409ba70 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient(hunting): picked mon.noname-c con 0x7f35a409beb0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a4072b00 con 0x7f35a409b690 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a4071f00 con 0x7f35a409ba70 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35a412bb30 con 0x7f35a409beb0 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a9516640 10 monclient(hunting): authenticate will time out at 1014.307376s 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a2ffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f35a409b690 0x7f35a40980b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55734/0 (socket says 172.21.15.6:55734) 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.395+0000 7f35a2ffd640 1 -- 172.21.15.6:0/921924201 learned_addr learned my addr 172.21.15.6:0/921924201 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a2ffd640 1 -- 172.21.15.6:0/921924201 >> v1:172.21.15.6:6789/0 conn(0x7f35a409b690 legacy=0x7f35a40980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a37fe640 1 -- 172.21.15.6:0/921924201 >> v1:172.21.15.6:6790/0 conn(0x7f35a409beb0 legacy=0x7f35a41d3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a2ffd640 1 --1- 172.21.15.6:0/921924201 >> v1:172.21.15.6:6789/0 conn(0x7f35a409b690 0x7f35a40980b0 :-1 s=OPENED pgs=222 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f35a4072b00 auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3207262792 0 0) 0x7f35a412bb30 con 0x7f35a409beb0 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): my global_id is 4553 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3584003530 con 0x7f35a409beb0 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a27fc640 1 -- 172.21.15.6:0/921924201 >> v1:172.21.15.203:6789/0 conn(0x7f35a409ba70 legacy=0x7f35a40987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1997391248 0 0) 0x7f3584003530 con 0x7f35a409beb0 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f35a412bb30 con 0x7f35a409beb0 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3594001b00 con 0x7f35a409beb0 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): dump: 2024-04-25T21:40:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 215338287 0 0) 0x7f35a412bb30 con 0x7f35a409beb0 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 >> v1:172.21.15.203:6789/0 conn(0x7f35a409ba70 legacy=0x7f35a40987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 >> v1:172.21.15.6:6789/0 conn(0x7f35a409b690 legacy=0x7f35a40980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 monclient: found mon.c 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a4098ed0 con 0x7f35a409beb0 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 5 monclient: authenticate success, global_id 4553 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 10 monclient: _renew_subs 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 275028097 0 0) 0x7f35a4072b00 con 0x7f35a409b690 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f35a409ed10 con 0x7f35a409beb0 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 10 monclient: _renew_subs 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f35a412b850 con 0x7f35a409beb0 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f35940038d0 con 0x7f35a409beb0 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f3594003c20 con 0x7f35a409beb0 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a17fa640 4 set_mon_vals no callback set 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 10 monclient: dump: 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3588005090 con 0x7f35a409beb0 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f3594003dc0 con 0x7f35a409beb0 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.399+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f35940442b0 con 0x7f35a409beb0 2024-04-25T21:40:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.403+0000 7f35a2ffd640 10 monclient: get_auth_request con 0x7f35840369d0 auth_method 0 2024-04-25T21:40:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.407+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f3594015690 con 0x7f35a409beb0 2024-04-25T21:40:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.407+0000 7f359b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.407+0000 7f359b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:55.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f3588000ba0 con 0x7f35840369d0 2024-04-25T21:40:55.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f359b7fe640 1 -- 172.21.15.6:0/921924201 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7f3588000ba0 con 0x7f35840369d0 2024-04-25T21:40:55.563 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T21:40:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 >> v1:172.21.15.203:6812/3164820986 conn(0x7f35840369d0 legacy=0x7f3584038e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 10 monclient: shutdown 2024-04-25T21:40:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 >> v1:172.21.15.6:6790/0 conn(0x7f35a409beb0 legacy=0x7f35a41d3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 shutdown_connections 2024-04-25T21:40:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 >> 172.21.15.6:0/921924201 conn(0x7f35a40a1c50 msgr2=0x7f35a41326d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 shutdown_connections 2024-04-25T21:40:55.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.563+0000 7f35a9516640 1 -- 172.21.15.6:0/921924201 wait complete. 2024-04-25T21:40:55.579 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T21:40:55.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient: get_monmap_and_config 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient: monmap: 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:55.701305+0000 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:55.701305+0000 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.700 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.701 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:55.701 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:55.701 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 1 Processor -- start 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 1 -- start start 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient: init 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient(hunting): picked mon.noname-b con 0x7fafec09c400 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient(hunting): picked mon.noname-a con 0x7fafec09c7e0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient(hunting): picked mon.noname-c con 0x7fafec139c10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafec0a0a10 con 0x7fafec09c7e0 2024-04-25T21:40:55.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafec09d810 con 0x7fafec139c10 2024-04-25T21:40:55.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafec09daf0 con 0x7fafec09c400 2024-04-25T21:40:55.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7faff108f640 10 monclient(hunting): authenticate will time out at 1014.611108s 2024-04-25T21:40:55.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7fafead9d640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fafec09c400 0x7fafec09ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40952/0 (socket says 172.21.15.6:40952) 2024-04-25T21:40:55.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.699+0000 7fafead9d640 1 -- 172.21.15.6:0/3147032738 learned_addr learned my addr 172.21.15.6:0/3147032738 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafead9d640 1 -- 172.21.15.6:0/3147032738 >> v1:172.21.15.6:6790/0 conn(0x7fafec09c400 legacy=0x7fafec09ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafea59c640 1 -- 172.21.15.6:0/3147032738 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c7e0 legacy=0x7fafec09cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1155155765 0 0) 0x7fafec09daf0 con 0x7fafec09c400 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): my global_id is 4556 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafd4003530 con 0x7fafec09c400 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 677249945 0 0) 0x7fafec0a0a10 con 0x7fafec09c7e0 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient(hunting): my global_id is 4593 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafec09daf0 con 0x7fafec09c7e0 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafeb59e640 1 -- 172.21.15.6:0/3147032738 >> v1:172.21.15.203:6789/0 conn(0x7fafec139c10 legacy=0x7fafec13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3207098452 0 0) 0x7fafd4003530 con 0x7fafec09c400 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 >> v1:172.21.15.203:6789/0 conn(0x7fafec139c10 legacy=0x7fafec13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c7e0 legacy=0x7fafec09cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 monclient: found mon.noname-b 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec0a2640 con 0x7fafec09c400 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafe4001c50 con 0x7fafec09c400 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 5 monclient: authenticate success, global_id 4556 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: dump: 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fafe4001fa0 con 0x7fafec09c400 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 1 -- 172.21.15.6:0/3147032738 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafe4004dd0 con 0x7fafec09c400 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7fafe9d9b640 10 monclient: dump: 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 4 set_mon_vals no callback set 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient: shutdown 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 1 -- 172.21.15.6:0/3147032738 >> v1:172.21.15.6:6790/0 conn(0x7fafec09c400 legacy=0x7fafec09ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 1 -- 172.21.15.6:0/3147032738 shutdown_connections 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 1 -- 172.21.15.6:0/3147032738 >> 172.21.15.6:0/3147032738 conn(0x7fafec0a5500 msgr2=0x7fafec0a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 1 -- 172.21.15.6:0/3147032738 shutdown_connections 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 1 -- 172.21.15.6:0/3147032738 wait complete. 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient: monmap: 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 1 Processor -- start 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 1 -- start start 2024-04-25T21:40:55.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient: init 2024-04-25T21:40:55.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient(hunting): picked mon.noname-a con 0x7fafec09c400 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient(hunting): picked mon.noname-c con 0x7fafec09c7e0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.703+0000 7faff108f640 10 monclient(hunting): picked mon.noname-b con 0x7fafec139c10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafec1349d0 con 0x7fafec09c400 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafec11f340 con 0x7fafec139c10 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafec11eaa0 con 0x7fafec09c7e0 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 10 monclient(hunting): authenticate will time out at 1014.616043s 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafead9d640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fafec09c400 0x7fafec1339a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55742/0 (socket says 172.21.15.6:55742) 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafead9d640 1 -- 172.21.15.6:0/4149767978 learned_addr learned my addr 172.21.15.6:0/4149767978 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafead9d640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c400 legacy=0x7fafec1339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafea59c640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6790/0 conn(0x7fafec09c7e0 legacy=0x7fafec1d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765359693 0 0) 0x7fafec0a2640 con 0x7fafec09c400 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): my global_id is 4596 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafd8003530 con 0x7fafec09c400 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431245178 0 0) 0x7fafec11eaa0 con 0x7fafec09c7e0 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): my global_id is 4559 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafec0a2640 con 0x7fafec09c7e0 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafeb59e640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.203:6789/0 conn(0x7fafec139c10 legacy=0x7fafec1d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 89774961 0 0) 0x7fafd8003530 con 0x7fafec09c400 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafec11eaa0 con 0x7fafec09c400 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafe4004d10 con 0x7fafec09c400 2024-04-25T21:40:55.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): dump: 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 782450351 0 0) 0x7fafec0a2640 con 0x7fafec09c7e0 2024-04-25T21:40:55.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fafd8003530 con 0x7fafec09c7e0 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafe0001be0 con 0x7fafec09c7e0 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient(hunting): dump: 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4284087254 0 0) 0x7fafec11eaa0 con 0x7fafec09c400 2024-04-25T21:40:55.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6790/0 conn(0x7fafec09c7e0 legacy=0x7fafec1d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.203:6789/0 conn(0x7fafec139c10 legacy=0x7fafec1d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 monclient: found mon.a 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec1340b0 con 0x7fafec09c400 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 5 monclient: authenticate success, global_id 4596 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 10 monclient: _renew_subs 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fafec11c560 con 0x7fafec09c400 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 10 monclient: _renew_subs 2024-04-25T21:40:55.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fafec1cbcd0 con 0x7fafec09c400 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fafe4001fa0 con 0x7fafec09c400 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafe4005520 con 0x7fafec09c400 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafe959a640 4 set_mon_vals no callback set 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 10 monclient: dump: 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.722 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fafcc000f90 con 0x7fafec09c400 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fafe40102c0 con 0x7fafec09c400 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafead9d640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c400 legacy=0x7fafec1339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafead9d640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c400 legacy=0x7fafec1339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafead9d640 1 --1- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c400 0x7fafec1339a0 :-1 s=OPENED pgs=224 cs=1 l=1).handle_message read tag failed 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.707+0000 7fafead9d640 1 --1- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c400 0x7fafec1339a0 :-1 s=OPENED pgs=224 cs=1 l=1).fault on lossy channel, failing 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.6:6789/0 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafec09c400 legacy=0x7fafec1339a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:55.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): picked mon.b con 0x7fafec139c10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): picked mon.c con 0x7fafec09c7e0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): picked mon.a con 0x7fafd8041da0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fafec11eaa0 con 0x7fafd8041da0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fafec0a2640 con 0x7fafec139c10 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fafd8047ef0 con 0x7fafec09c7e0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafead9d640 10 monclient(hunting): get_auth_request con 0x7fafd8036b40 auth_method 0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafeb59e640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6790/0 conn(0x7fafec09c7e0 legacy=0x7fafd8040240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafead9d640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafd8041da0 legacy=0x7fafd8044230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429602002 0 0) 0x7fafd8047ef0 con 0x7fafec09c7e0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fafd8048290 con 0x7fafec09c7e0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158679884 0 0) 0x7fafec11eaa0 con 0x7fafd8041da0 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fafd8047ef0 con 0x7fafd8041da0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafea59c640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.203:6789/0 conn(0x7fafec139c10 legacy=0x7fafd803c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1120171481 0 0) 0x7fafd8048290 con 0x7fafec09c7e0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.203:6789/0 conn(0x7fafec139c10 legacy=0x7fafd803c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6789/0 conn(0x7fafd8041da0 legacy=0x7fafd8044230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 monclient: found mon.c 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fafd80490f0 con 0x7fafec09c7e0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=2) -- 0x7fafd8049320 con 0x7fafec09c7e0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=3) -- 0x7fafd80480c0 con 0x7fafec09c7e0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=4) -- 0x7fafd8048520 con 0x7fafec09c7e0 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fafdc001950 con 0x7fafec09c7e0 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: dump: 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.726 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fafd8048520 con 0x7fafec09c7e0 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafe959a640 4 set_mon_vals no callback set 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fafdc001770 con 0x7fafec09c7e0 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 6 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fafdc0435a0 con 0x7fafec09c7e0 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.203:6800/3201437079 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafd8046cc0 con 0x7fafec09c400 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mon_get_version_reply(handle=2 version=25) ==== 24+0+0 (unknown 1434485932 0 0) 0x7fafdc0438e0 con 0x7fafec09c7e0 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: handle_get_version_reply finishing 2 version 25 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 8 ==== mon_get_version_reply(handle=3 version=25) ==== 24+0+0 (unknown 4023338530 0 0) 0x7fafdc043ba0 con 0x7fafec09c7e0 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: handle_get_version_reply finishing 3 version 25 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=4 version=25) ==== 24+0+0 (unknown 3323236507 0 0) 0x7fafdc043e60 con 0x7fafec09c7e0 2024-04-25T21:40:55.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.711+0000 7fafd37fe640 10 monclient: handle_get_version_reply finishing 4 version 25 2024-04-25T21:40:55.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.715+0000 7fafea59c640 10 monclient: get_auth_request con 0x7fafec09c400 auth_method 0 2024-04-25T21:40:55.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.719+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== osd.3 v1:172.21.15.203:6800/3201437079 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fafd8046cc0 con 0x7fafec09c400 2024-04-25T21:40:55.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.731+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 --> v1:172.21.15.203:6800/3201437079 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fafcc002cc0 con 0x7fafec09c400 2024-04-25T21:40:55.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.731+0000 7fafd37fe640 1 -- 172.21.15.6:0/4149767978 <== osd.3 v1:172.21.15.203:6800/3201437079 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fafcc002cc0 con 0x7fafec09c400 2024-04-25T21:40:55.734 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T21:40:55.734 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:55.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.203:6800/3201437079 conn(0x7fafec09c400 legacy=0x7fafd804b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.203:6812/3164820986 conn(0x7fafd8036b40 legacy=0x7fafd8039000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 10 monclient: shutdown 2024-04-25T21:40:55.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 >> v1:172.21.15.6:6790/0 conn(0x7fafec09c7e0 legacy=0x7fafd8040240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7fafeb59e640 1 -- 172.21.15.6:0/4149767978 reap_dead start 2024-04-25T21:40:55.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 shutdown_connections 2024-04-25T21:40:55.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 >> 172.21.15.6:0/4149767978 conn(0x7fafec0a5500 msgr2=0x7fafec071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 shutdown_connections 2024-04-25T21:40:55.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.735+0000 7faff108f640 1 -- 172.21.15.6:0/4149767978 wait complete. 2024-04-25T21:40:55.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.851+0000 7fbecbe22640 10 monclient: get_monmap_and_config 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.851+0000 7fbecbe22640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.851+0000 7fbecbe22640 10 monclient: monmap: 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:55.854433+0000 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:55.854433+0000 2024-04-25T21:40:55.853 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:55.854 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:55.854 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.851+0000 7fbecbe22640 1 Processor -- start 2024-04-25T21:40:55.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.851+0000 7fbecbe22640 1 -- start start 2024-04-25T21:40:55.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.851+0000 7fbecbe22640 10 monclient: init 2024-04-25T21:40:55.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient(hunting): picked mon.noname-c con 0x7fbec40721d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient(hunting): picked mon.noname-b con 0x7fbec4136d30 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient(hunting): picked mon.noname-a con 0x7fbec413d290 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec40a2130 con 0x7fbec413d290 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec406e650 con 0x7fbec40721d0 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4072e50 con 0x7fbec4136d30 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient(hunting): authenticate will time out at 1014.764458s 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbeca3bf640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbec413d290 0x7fbec413f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55746/0 (socket says 172.21.15.6:55746) 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec93bd640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbec4136d30 0x7fbec413bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40980/0 (socket says 172.21.15.6:40980) 2024-04-25T21:40:55.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbeca3bf640 1 -- 172.21.15.6:0/1612767799 learned_addr learned my addr 172.21.15.6:0/1612767799 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbeca3bf640 1 -- 172.21.15.6:0/1612767799 >> v1:172.21.15.6:6789/0 conn(0x7fbec413d290 legacy=0x7fbec413f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec93bd640 1 -- 172.21.15.6:0/1612767799 >> v1:172.21.15.6:6790/0 conn(0x7fbec4136d30 legacy=0x7fbec413bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec9bbe640 1 -- 172.21.15.6:0/1612767799 >> v1:172.21.15.203:6789/0 conn(0x7fbec40721d0 legacy=0x7fbec40725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167195022 0 0) 0x7fbec4072e50 con 0x7fbec4136d30 2024-04-25T21:40:55.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): my global_id is 4562 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeac003530 con 0x7fbec4136d30 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062740090 0 0) 0x7fbec40a2130 con 0x7fbec413d290 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient(hunting): my global_id is 4599 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec4072e50 con 0x7fbec413d290 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 81827065 0 0) 0x7fbec4072e50 con 0x7fbec413d290 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 >> v1:172.21.15.6:6790/0 conn(0x7fbec4136d30 legacy=0x7fbec413bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 >> v1:172.21.15.203:6789/0 conn(0x7fbec40721d0 legacy=0x7fbec40725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 monclient: found mon.noname-a 2024-04-25T21:40:55.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec4137180 con 0x7fbec413d290 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbebc001990 con 0x7fbec413d290 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: dump: 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.862 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 5 monclient: authenticate success, global_id 4599 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbebc001cb0 con 0x7fbec413d290 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 1 -- 172.21.15.6:0/1612767799 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbebc004a50 con 0x7fbec413d290 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbec8bbc640 10 monclient: dump: 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.863 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.855+0000 7fbecbe22640 4 set_mon_vals no callback set 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient: shutdown 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- 172.21.15.6:0/1612767799 >> v1:172.21.15.6:6789/0 conn(0x7fbec413d290 legacy=0x7fbec413f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- 172.21.15.6:0/1612767799 shutdown_connections 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- 172.21.15.6:0/1612767799 >> 172.21.15.6:0/1612767799 conn(0x7fbec40a5f90 msgr2=0x7fbec40a63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- 172.21.15.6:0/1612767799 shutdown_connections 2024-04-25T21:40:55.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- 172.21.15.6:0/1612767799 wait complete. 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient: build_initial_monmap 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient: monmap: 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 Processor -- start 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- start start 2024-04-25T21:40:55.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient: init 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient(hunting): picked mon.noname-c con 0x7fbec40721d0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient(hunting): picked mon.noname-a con 0x7fbec4136d30 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient(hunting): picked mon.noname-b con 0x7fbec413d290 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4138620 con 0x7fbec4136d30 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec40a14d0 con 0x7fbec413d290 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4135f40 con 0x7fbec40721d0 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbecbe22640 10 monclient(hunting): authenticate will time out at 1014.769628s 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbec9bbe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fbec40721d0 0x7fbec4137800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40988/0 (socket says 172.21.15.6:40988) 2024-04-25T21:40:55.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbec9bbe640 1 -- 172.21.15.6:0/2410236060 learned_addr learned my addr 172.21.15.6:0/2410236060 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbec9bbe640 1 -- 172.21.15.6:0/2410236060 >> v1:172.21.15.6:6790/0 conn(0x7fbec40721d0 legacy=0x7fbec4137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.859+0000 7fbec93bd640 1 -- 172.21.15.6:0/2410236060 >> v1:172.21.15.6:6789/0 conn(0x7fbec4136d30 legacy=0x7fbec4137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126478077 0 0) 0x7fbec4135f40 con 0x7fbec40721d0 2024-04-25T21:40:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): my global_id is 4565 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb0003530 con 0x7fbec40721d0 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574680012 0 0) 0x7fbec4138620 con 0x7fbec4136d30 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): my global_id is 4602 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec4135f40 con 0x7fbec4136d30 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbeca3bf640 1 -- 172.21.15.6:0/2410236060 >> v1:172.21.15.203:6789/0 conn(0x7fbec413d290 legacy=0x7fbec41d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1011864231 0 0) 0x7fbeb0003530 con 0x7fbec40721d0 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbec4138620 con 0x7fbec40721d0 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbeb4001f40 con 0x7fbec40721d0 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:55.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1955629772 0 0) 0x7fbec4135f40 con 0x7fbec4136d30 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbeb0003530 con 0x7fbec4136d30 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbeb80047f0 con 0x7fbec4136d30 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): dump: 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3787953540 0 0) 0x7fbec40a14d0 con 0x7fbec413d290 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _negotiate 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient(hunting): my global_id is 94198 2024-04-25T21:40:55.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec4135f40 con 0x7fbec413d290 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3598607613 0 0) 0x7fbec4138620 con 0x7fbec40721d0 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 >> v1:172.21.15.203:6789/0 conn(0x7fbec413d290 legacy=0x7fbec41d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 >> v1:172.21.15.6:6789/0 conn(0x7fbec4136d30 legacy=0x7fbec4137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: _finish_hunting 0 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 monclient: found mon.c 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec413b3a0 con 0x7fbec40721d0 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: _finish_auth 0 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: _check_auth_tickets 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 5 monclient: authenticate success, global_id 4565 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 10 monclient: _renew_subs 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbec4139c30 con 0x7fbec40721d0 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 10 monclient: _renew_subs 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbec4139ef0 con 0x7fbec40721d0 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbeb4003d30 con 0x7fbec40721d0 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fbeb4004080 con 0x7fbec40721d0 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec3fff640 4 set_mon_vals no callback set 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 10 monclient: dump: 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.874 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea4005090 con 0x7fbec40721d0 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fbeb4004260 con 0x7fbec40721d0 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fbeb4044790 con 0x7fbec40721d0 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.863+0000 7fbec93bd640 10 monclient: get_auth_request con 0x7fbeb0036ce0 auth_method 0 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.867+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fbeb4015b90 con 0x7fbec40721d0 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.867+0000 7fbec27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:55.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:55.867+0000 7fbec27fc640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.023+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fbea4000ba0 con 0x7fbeb0036ce0 2024-04-25T21:40:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.023+0000 7fbec27fc640 1 -- 172.21.15.6:0/2410236060 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7fbea4000ba0 con 0x7fbeb0036ce0 2024-04-25T21:40:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 >> v1:172.21.15.203:6812/3164820986 conn(0x7fbeb0036ce0 legacy=0x7fbeb00391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 10 monclient: shutdown 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 >> v1:172.21.15.6:6790/0 conn(0x7fbec40721d0 legacy=0x7fbec4137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 shutdown_connections 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 >> 172.21.15.6:0/2410236060 conn(0x7fbec40a5f90 msgr2=0x7fbec4070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 shutdown_connections 2024-04-25T21:40:56.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.027+0000 7fbecbe22640 1 -- 172.21.15.6:0/2410236060 wait complete. 2024-04-25T21:40:56.042 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T21:40:56.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T21:40:56.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.163+0000 7fe5f04fd640 10 monclient: get_monmap_and_config 2024-04-25T21:40:56.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.163+0000 7fe5f04fd640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient: monmap: 2024-04-25T21:40:56.166 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.166 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.166 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:56.167685+0000 2024-04-25T21:40:56.167 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:56.167685+0000 2024-04-25T21:40:56.167 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.167 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.167 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.167 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:56.167 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:56.167 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 1 Processor -- start 2024-04-25T21:40:56.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 1 -- start start 2024-04-25T21:40:56.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient: init 2024-04-25T21:40:56.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:40:56.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e809c400 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e809c7e0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e8139c10 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e80a0a10 con 0x7fe5e809c400 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e809d810 con 0x7fe5e8139c10 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e809daf0 con 0x7fe5e809c7e0 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5f04fd640 10 monclient(hunting): authenticate will time out at 1015.077510s 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5ee299640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe5e809c400 0x7fe5e809ea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55758/0 (socket says 172.21.15.6:55758) 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5ee299640 1 -- 172.21.15.6:0/1472312946 learned_addr learned my addr 172.21.15.6:0/1472312946 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5eda98640 1 --1- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6790/0 conn(0x7fe5e809c7e0 0x7fe5e809cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:40992/0 (socket says 172.21.15.6:40992) 2024-04-25T21:40:56.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.167+0000 7fe5ee299640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6789/0 conn(0x7fe5e809c400 legacy=0x7fe5e809ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5eda98640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6790/0 conn(0x7fe5e809c7e0 legacy=0x7fe5e809cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ee299640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6789/0 conn(0x7fe5e809c400 legacy=0x7fe5e809ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T21:40:56.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ee299640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6789/0 conn(0x7fe5e809c400 legacy=0x7fe5e809ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ee299640 1 --1- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6789/0 conn(0x7fe5e809c400 0x7fe5e809ea90 :-1 s=OPENED pgs=230 cs=1 l=1).handle_message read tag failed 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ee299640 1 --1- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6789/0 conn(0x7fe5e809c400 0x7fe5e809ea90 :-1 s=OPENED pgs=230 cs=1 l=1).fault on lossy channel, failing 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.6:6789/0 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5eea9a640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.203:6789/0 conn(0x7fe5e8139c10 legacy=0x7fe5e813c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464938640 0 0) 0x7fe5e809daf0 con 0x7fe5e809c7e0 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): my global_id is 4568 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5d4003530 con 0x7fe5e809c7e0 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596229647 0 0) 0x7fe5e809d810 con 0x7fe5e8139c10 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient(hunting): my global_id is 94201 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5e809daf0 con 0x7fe5e8139c10 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1188761738 0 0) 0x7fe5d4003530 con 0x7fe5e809c7e0 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.203:6789/0 conn(0x7fe5e8139c10 legacy=0x7fe5e813c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6789/0 conn(0x7fe5e809c400 legacy=0x7fe5e809ea90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 monclient: found mon.noname-b 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e80a2640 con 0x7fe5e809c7e0 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe5d80018d0 con 0x7fe5e809c7e0 2024-04-25T21:40:56.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 5 monclient: authenticate success, global_id 4568 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: dump: 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5d8001c20 con 0x7fe5e809c7e0 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 1 -- 172.21.15.6:0/1472312946 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe5d8004990 con 0x7fe5e809c7e0 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 4 set_mon_vals no callback set 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5ed297640 10 monclient: dump: 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient: shutdown 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- 172.21.15.6:0/1472312946 >> v1:172.21.15.6:6790/0 conn(0x7fe5e809c7e0 legacy=0x7fe5e809cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- 172.21.15.6:0/1472312946 shutdown_connections 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- 172.21.15.6:0/1472312946 >> 172.21.15.6:0/1472312946 conn(0x7fe5e80a5500 msgr2=0x7fe5e80a1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- 172.21.15.6:0/1472312946 shutdown_connections 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- 172.21.15.6:0/1472312946 wait complete. 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient: monmap: 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 Processor -- start 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- start start 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient: init 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient(hunting): picked mon.noname-c con 0x7fe5e809c400 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient(hunting): picked mon.noname-b con 0x7fe5e809c7e0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient(hunting): picked mon.noname-a con 0x7fe5e8139c10 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e80715c0 con 0x7fe5e8139c10 2024-04-25T21:40:56.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e811c910 con 0x7fe5e809c7e0 2024-04-25T21:40:56.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5e811cbf0 con 0x7fe5e809c400 2024-04-25T21:40:56.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5f04fd640 10 monclient(hunting): authenticate will time out at 1015.082947s 2024-04-25T21:40:56.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5eea9a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe5e8139c10 0x7fe5e81d1f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55768/0 (socket says 172.21.15.6:55768) 2024-04-25T21:40:56.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.171+0000 7fe5eea9a640 1 -- 172.21.15.6:0/908467009 learned_addr learned my addr 172.21.15.6:0/908467009 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5eea9a640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.6:6789/0 conn(0x7fe5e8139c10 legacy=0x7fe5e81d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5ee299640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.6:6790/0 conn(0x7fe5e809c400 legacy=0x7fe5e80707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1270756294 0 0) 0x7fe5e811cbf0 con 0x7fe5e809c400 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): my global_id is 4571 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5cc003530 con 0x7fe5e809c400 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3411672402 0 0) 0x7fe5e80715c0 con 0x7fe5e8139c10 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): my global_id is 4605 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5e811cbf0 con 0x7fe5e8139c10 2024-04-25T21:40:56.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5eda98640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.203:6789/0 conn(0x7fe5e809c7e0 legacy=0x7fe5e8070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2280874548 0 0) 0x7fe5cc003530 con 0x7fe5e809c400 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5e80715c0 con 0x7fe5e809c400 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe5e4001ab0 con 0x7fe5e809c400 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): dump: 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.184 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1914342929 0 0) 0x7fe5e811cbf0 con 0x7fe5e8139c10 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe5cc003530 con 0x7fe5e8139c10 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe5e0001f60 con 0x7fe5e8139c10 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): dump: 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.185 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3749624360 0 0) 0x7fe5e811c910 con 0x7fe5e809c7e0 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient(hunting): my global_id is 94204 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5e811cbf0 con 0x7fe5e809c7e0 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 296082096 0 0) 0x7fe5e80715c0 con 0x7fe5e809c400 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.203:6789/0 conn(0x7fe5e809c7e0 legacy=0x7fe5e8070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.6:6789/0 conn(0x7fe5e8139c10 legacy=0x7fe5e81d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 monclient: found mon.c 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e811b6e0 con 0x7fe5e809c400 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 5 monclient: authenticate success, global_id 4571 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 10 monclient: _renew_subs 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e8133e70 con 0x7fe5e809c400 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 10 monclient: _renew_subs 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe5e811b9b0 con 0x7fe5e809c400 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe5e4004b20 con 0x7fe5e809c400 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fe5e4004e70 con 0x7fe5e809c400 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:56.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5eca96640 4 set_mon_vals no callback set 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: dump: 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe5d0000f90 con 0x7fe5e809c400 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fe5e4005090 con 0x7fe5e809c400 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fe5e4044380 con 0x7fe5e809c400 2024-04-25T21:40:56.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.203:6801/1405305648 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5cc0415a0 con 0x7fe5cc03d930 2024-04-25T21:40:56.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (unknown 2663167439 0 0) 0x7fe5e40447b0 con 0x7fe5e809c400 2024-04-25T21:40:56.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5deffd640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-04-25T21:40:56.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5eda98640 10 monclient: get_auth_request con 0x7fe5cc036ce0 auth_method 0 2024-04-25T21:40:56.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.175+0000 7fe5eea9a640 10 monclient: get_auth_request con 0x7fe5cc03d930 auth_method 0 2024-04-25T21:40:56.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.183+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== osd.4 v1:172.21.15.203:6801/1405305648 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe5cc0415a0 con 0x7fe5cc03d930 2024-04-25T21:40:56.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.195+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 --> v1:172.21.15.203:6801/1405305648 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe5d0002cc0 con 0x7fe5cc03d930 2024-04-25T21:40:56.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.195+0000 7fe5deffd640 1 -- 172.21.15.6:0/908467009 <== osd.4 v1:172.21.15.203:6801/1405305648 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fe5d0002cc0 con 0x7fe5cc03d930 2024-04-25T21:40:56.198 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:56.198 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T21:40:56.198 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:56.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.203:6801/1405305648 conn(0x7fe5cc03d930 legacy=0x7fe5cc03fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.203:6812/3164820986 conn(0x7fe5cc036ce0 legacy=0x7fe5cc0391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 10 monclient: shutdown 2024-04-25T21:40:56.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 >> v1:172.21.15.6:6790/0 conn(0x7fe5e809c400 legacy=0x7fe5e80707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5eea9a640 1 -- 172.21.15.6:0/908467009 reap_dead start 2024-04-25T21:40:56.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 shutdown_connections 2024-04-25T21:40:56.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 >> 172.21.15.6:0/908467009 conn(0x7fe5e80a5500 msgr2=0x7fe5e80a3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 shutdown_connections 2024-04-25T21:40:56.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.199+0000 7fe5f04fd640 1 -- 172.21.15.6:0/908467009 wait complete. 2024-04-25T21:40:56.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient: get_monmap_and_config 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient: monmap: 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:56.332646+0000 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:56.332646+0000 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.331 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.332 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.332 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:56.332 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:56.332 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 1 Processor -- start 2024-04-25T21:40:56.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 1 -- start start 2024-04-25T21:40:56.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient: init 2024-04-25T21:40:56.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:56.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient(hunting): picked mon.noname-a con 0x7f16000a2c20 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient(hunting): picked mon.noname-c con 0x7f16000a40d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient(hunting): picked mon.noname-b con 0x7f160013d580 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16000a0e50 con 0x7f16000a2c20 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16000a1130 con 0x7f16000a40d0 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16000a3710 con 0x7f160013d580 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f1605801640 10 monclient(hunting): authenticate will time out at 1015.242442s 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f15feffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f16000a2c20 0x7f16000a3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55776/0 (socket says 172.21.15.6:55776) 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f15ff7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f160013d580 0x7f160013f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41010/0 (socket says 172.21.15.6:41010) 2024-04-25T21:40:56.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.331+0000 7f15feffd640 1 -- 172.21.15.6:0/2911253202 learned_addr learned my addr 172.21.15.6:0/2911253202 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15feffd640 1 -- 172.21.15.6:0/2911253202 >> v1:172.21.15.6:6789/0 conn(0x7f16000a2c20 legacy=0x7f16000a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15ff7fe640 1 -- 172.21.15.6:0/2911253202 >> v1:172.21.15.6:6790/0 conn(0x7f160013d580 legacy=0x7f160013f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3976065308 0 0) 0x7f16000a0e50 con 0x7f16000a2c20 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): my global_id is 4608 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15e8003530 con 0x7f16000a2c20 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2414355030 0 0) 0x7f16000a3710 con 0x7f160013d580 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient(hunting): my global_id is 4574 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16000a0e50 con 0x7f160013d580 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fe7fc640 1 -- 172.21.15.6:0/2911253202 >> v1:172.21.15.203:6789/0 conn(0x7f16000a40d0 legacy=0x7f160013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1754086035 0 0) 0x7f15e8003530 con 0x7f16000a2c20 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 >> v1:172.21.15.6:6790/0 conn(0x7f160013d580 legacy=0x7f160013f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 >> v1:172.21.15.203:6789/0 conn(0x7f16000a40d0 legacy=0x7f160013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 monclient: found mon.noname-a 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160006e9f0 con 0x7f16000a2c20 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15f00018d0 con 0x7f16000a2c20 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: dump: 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 5 monclient: authenticate success, global_id 4608 2024-04-25T21:40:56.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:56.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15f0001c20 con 0x7f16000a2c20 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 1 -- 172.21.15.6:0/2911253202 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15f0004a50 con 0x7f16000a2c20 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15fdffb640 10 monclient: dump: 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.341 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 4 set_mon_vals no callback set 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient: shutdown 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- 172.21.15.6:0/2911253202 >> v1:172.21.15.6:6789/0 conn(0x7f16000a2c20 legacy=0x7f16000a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- 172.21.15.6:0/2911253202 shutdown_connections 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- 172.21.15.6:0/2911253202 >> 172.21.15.6:0/2911253202 conn(0x7f16000a5e10 msgr2=0x7f16000a6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- 172.21.15.6:0/2911253202 shutdown_connections 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- 172.21.15.6:0/2911253202 wait complete. 2024-04-25T21:40:56.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient: monmap: 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 Processor -- start 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- start start 2024-04-25T21:40:56.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient: init 2024-04-25T21:40:56.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient(hunting): picked mon.noname-a con 0x7f16000a2c20 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient(hunting): picked mon.noname-b con 0x7f16000a40d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient(hunting): picked mon.noname-c con 0x7f160013d580 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16001406f0 con 0x7f16000a2c20 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16001221d0 con 0x7f16000a40d0 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16001244c0 con 0x7f160013d580 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f1605801640 10 monclient(hunting): authenticate will time out at 1015.247485s 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15feffd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f16000a2c20 0x7f160013cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55792/0 (socket says 172.21.15.6:55792) 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.335+0000 7f15feffd640 1 -- 172.21.15.6:0/2956813566 learned_addr learned my addr 172.21.15.6:0/2956813566 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15ff7fe640 1 -- 172.21.15.6:0/2956813566 >> v1:172.21.15.6:6790/0 conn(0x7f160013d580 legacy=0x7f16001d4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15feffd640 1 -- 172.21.15.6:0/2956813566 >> v1:172.21.15.6:6789/0 conn(0x7f16000a2c20 legacy=0x7f160013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 488243688 0 0) 0x7f16001406f0 con 0x7f16000a2c20 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): my global_id is 4611 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15ec003530 con 0x7f16000a2c20 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845301569 0 0) 0x7f16001244c0 con 0x7f160013d580 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): my global_id is 4577 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16001406f0 con 0x7f160013d580 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15fe7fc640 1 -- 172.21.15.6:0/2956813566 >> v1:172.21.15.203:6789/0 conn(0x7f16000a40d0 legacy=0x7f16001d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2926332654 0 0) 0x7f16001406f0 con 0x7f160013d580 2024-04-25T21:40:56.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16001244c0 con 0x7f160013d580 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15f8001fc0 con 0x7f160013d580 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): dump: 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2329768815 0 0) 0x7f15ec003530 con 0x7f16000a2c20 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f16001406f0 con 0x7f16000a2c20 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15f0004990 con 0x7f16000a2c20 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient(hunting): dump: 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.350 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 898021679 0 0) 0x7f16001244c0 con 0x7f160013d580 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 >> v1:172.21.15.203:6789/0 conn(0x7f16000a40d0 legacy=0x7f16001d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 >> v1:172.21.15.6:6789/0 conn(0x7f16000a2c20 legacy=0x7f160013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 monclient: found mon.c 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1600123c20 con 0x7f160013d580 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 5 monclient: authenticate success, global_id 4577 2024-04-25T21:40:56.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 10 monclient: _renew_subs 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1600123f00 con 0x7f160013d580 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 10 monclient: _renew_subs 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f16001cb990 con 0x7f160013d580 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15f8004e20 con 0x7f160013d580 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f15f8005170 con 0x7f160013d580 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15fd7fa640 4 set_mon_vals no callback set 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 10 monclient: dump: 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f15f8005390 con 0x7f160013d580 2024-04-25T21:40:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e0005090 con 0x7f160013d580 2024-04-25T21:40:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f15f8044760 con 0x7f160013d580 2024-04-25T21:40:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.339+0000 7f15feffd640 10 monclient: get_auth_request con 0x7f15ec036ac0 auth_method 0 2024-04-25T21:40:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.343+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f15f8015a80 con 0x7f160013d580 2024-04-25T21:40:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.343+0000 7f15e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.343+0000 7f15e77fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:56.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f15e0000ba0 con 0x7f15ec036ac0 2024-04-25T21:40:56.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f15e77fe640 1 -- 172.21.15.6:0/2956813566 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f15e0000ba0 con 0x7f15ec036ac0 2024-04-25T21:40:56.499 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T21:40:56.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 >> v1:172.21.15.203:6812/3164820986 conn(0x7f15ec036ac0 legacy=0x7f15ec038f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 10 monclient: shutdown 2024-04-25T21:40:56.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 >> v1:172.21.15.6:6790/0 conn(0x7f160013d580 legacy=0x7f16001d4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 shutdown_connections 2024-04-25T21:40:56.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 >> 172.21.15.6:0/2956813566 conn(0x7f16000a5e10 msgr2=0x7f1600140900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 shutdown_connections 2024-04-25T21:40:56.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.499+0000 7f1605801640 1 -- 172.21.15.6:0/2956813566 wait complete. 2024-04-25T21:40:56.515 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T21:40:56.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T21:40:56.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient: get_monmap_and_config 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient: monmap: 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:56.643964+0000 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:56.643964+0000 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:56.643 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 1 Processor -- start 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 1 -- start start 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient: init 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient(hunting): picked mon.noname-a con 0x7f7e4809a1d0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient(hunting): picked mon.noname-c con 0x7f7e4809c5f0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient(hunting): picked mon.noname-b con 0x7f7e4809acc0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e4809bd80 con 0x7f7e4809a1d0 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e480a2550 con 0x7f7e4809c5f0 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e4809b750 con 0x7f7e4809acc0 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e4d485640 10 monclient(hunting): authenticate will time out at 1015.554376s 2024-04-25T21:40:56.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e477fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7e4809acc0 0x7f7e4813c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41018/0 (socket says 172.21.15.6:41018) 2024-04-25T21:40:56.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e477fe640 1 -- 172.21.15.6:0/1203622420 learned_addr learned my addr 172.21.15.6:0/1203622420 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.643+0000 7f7e46ffd640 1 --1- 172.21.15.6:0/1203622420 >> v1:172.21.15.6:6789/0 conn(0x7f7e4809a1d0 0x7f7e4809a5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55798/0 (socket says 172.21.15.6:55798) 2024-04-25T21:40:56.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e477fe640 1 -- 172.21.15.6:0/1203622420 >> v1:172.21.15.6:6790/0 conn(0x7f7e4809acc0 legacy=0x7f7e4813c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e46ffd640 1 -- 172.21.15.6:0/1203622420 >> v1:172.21.15.6:6789/0 conn(0x7f7e4809a1d0 legacy=0x7f7e4809a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e467fc640 1 -- 172.21.15.6:0/1203622420 >> v1:172.21.15.203:6789/0 conn(0x7f7e4809c5f0 legacy=0x7f7e48138ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185482596 0 0) 0x7f7e4809b750 con 0x7f7e4809acc0 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): my global_id is 4580 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e30003530 con 0x7f7e4809acc0 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3409093816 0 0) 0x7f7e4809bd80 con 0x7f7e4809a1d0 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): my global_id is 4614 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e4809b750 con 0x7f7e4809a1d0 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775350416 0 0) 0x7f7e480a2550 con 0x7f7e4809c5f0 2024-04-25T21:40:56.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient(hunting): my global_id is 94213 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e4809bd80 con 0x7f7e4809c5f0 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4094772767 0 0) 0x7f7e30003530 con 0x7f7e4809acc0 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 >> v1:172.21.15.203:6789/0 conn(0x7f7e4809c5f0 legacy=0x7f7e48138ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 >> v1:172.21.15.6:6789/0 conn(0x7f7e4809a1d0 legacy=0x7f7e4809a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 monclient: found mon.noname-b 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4809ba30 con 0x7f7e4809acc0 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7e40001c50 con 0x7f7e4809acc0 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 5 monclient: authenticate success, global_id 4580 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: dump: 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.652 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e40001fa0 con 0x7f7e4809acc0 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 1 -- 172.21.15.6:0/1203622420 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7e40004dd0 con 0x7f7e4809acc0 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e45ffb640 10 monclient: dump: 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.653 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 4 set_mon_vals no callback set 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient: shutdown 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- 172.21.15.6:0/1203622420 >> v1:172.21.15.6:6790/0 conn(0x7f7e4809acc0 legacy=0x7f7e4813c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- 172.21.15.6:0/1203622420 shutdown_connections 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- 172.21.15.6:0/1203622420 >> 172.21.15.6:0/1203622420 conn(0x7f7e480a2e30 msgr2=0x7f7e480a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- 172.21.15.6:0/1203622420 shutdown_connections 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- 172.21.15.6:0/1203622420 wait complete. 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient: monmap: 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 Processor -- start 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- start start 2024-04-25T21:40:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient: init 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient(hunting): picked mon.noname-b con 0x7f7e4809a1d0 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient(hunting): picked mon.noname-a con 0x7f7e4809acc0 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient(hunting): picked mon.noname-c con 0x7f7e4809c5f0 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e480914e0 con 0x7f7e4809acc0 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e48133080 con 0x7f7e4809a1d0 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e48133360 con 0x7f7e4809c5f0 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e4d485640 10 monclient(hunting): authenticate will time out at 1015.559872s 2024-04-25T21:40:56.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e467fc640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7e4809acc0 0x7f7e48090dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55812/0 (socket says 172.21.15.6:55812) 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e467fc640 1 -- 172.21.15.6:0/2525501659 learned_addr learned my addr 172.21.15.6:0/2525501659 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.647+0000 7f7e477fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f7e4809c5f0 0x7f7e481d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41026/0 (socket says 172.21.15.6:41026) 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e477fe640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.6:6790/0 conn(0x7f7e4809c5f0 legacy=0x7f7e481d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e467fc640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.6:6789/0 conn(0x7f7e4809acc0 legacy=0x7f7e48090dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530749744 0 0) 0x7f7e48133360 con 0x7f7e4809c5f0 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): my global_id is 4583 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e34003530 con 0x7f7e4809c5f0 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994470336 0 0) 0x7f7e480914e0 con 0x7f7e4809acc0 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): my global_id is 4617 2024-04-25T21:40:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e48133360 con 0x7f7e4809acc0 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e46ffd640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.203:6789/0 conn(0x7f7e4809a1d0 legacy=0x7f7e480906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3349146695 0 0) 0x7f7e34003530 con 0x7f7e4809c5f0 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e480914e0 con 0x7f7e4809c5f0 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7e40004d10 con 0x7f7e4809c5f0 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): dump: 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.661 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3671473894 0 0) 0x7f7e48133360 con 0x7f7e4809acc0 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7e34003530 con 0x7f7e4809acc0 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7e3c001c50 con 0x7f7e4809acc0 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): dump: 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234335985 0 0) 0x7f7e48133080 con 0x7f7e4809a1d0 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient(hunting): my global_id is 94216 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e48133360 con 0x7f7e4809a1d0 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 667055601 0 0) 0x7f7e480914e0 con 0x7f7e4809c5f0 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.203:6789/0 conn(0x7f7e4809a1d0 legacy=0x7f7e480906c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.6:6789/0 conn(0x7f7e4809acc0 legacy=0x7f7e48090dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 monclient: found mon.c 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e481344d0 con 0x7f7e4809c5f0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 5 monclient: authenticate success, global_id 4583 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 10 monclient: _renew_subs 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7e48132c30 con 0x7f7e4809c5f0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 10 monclient: _renew_subs 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7e48133f90 con 0x7f7e4809c5f0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7e40001f20 con 0x7f7e4809c5f0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f7e40005560 con 0x7f7e4809c5f0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e457fa640 4 set_mon_vals no callback set 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 10 monclient: dump: 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.6:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7e28000f90 con 0x7f7e4809c5f0 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.651+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f7e40010300 con 0x7f7e4809c5f0 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.655+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f7e40044750 con 0x7f7e4809c5f0 2024-04-25T21:40:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.655+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.203:6808/1687281622 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e340414b0 con 0x7f7e3403d840 2024-04-25T21:40:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.655+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (unknown 2663167439 0 0) 0x7f7e40044b80 con 0x7f7e4809c5f0 2024-04-25T21:40:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.655+0000 7f7e2f7fe640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-04-25T21:40:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.655+0000 7f7e46ffd640 10 monclient: get_auth_request con 0x7f7e34036bf0 auth_method 0 2024-04-25T21:40:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.655+0000 7f7e467fc640 10 monclient: get_auth_request con 0x7f7e3403d840 auth_method 0 2024-04-25T21:40:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.659+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== osd.5 v1:172.21.15.203:6808/1687281622 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7e340414b0 con 0x7f7e3403d840 2024-04-25T21:40:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 --> v1:172.21.15.203:6808/1687281622 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7e28002cc0 con 0x7f7e3403d840 2024-04-25T21:40:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e2f7fe640 1 -- 172.21.15.6:0/2525501659 <== osd.5 v1:172.21.15.203:6808/1687281622 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f7e28002cc0 con 0x7f7e3403d840 2024-04-25T21:40:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-04-25T21:40:56.675 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T21:40:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.203:6808/1687281622 conn(0x7f7e3403d840 legacy=0x7f7e3403fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.203:6812/3164820986 conn(0x7f7e34036bf0 legacy=0x7f7e340390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 10 monclient: shutdown 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 >> v1:172.21.15.6:6790/0 conn(0x7f7e4809c5f0 legacy=0x7f7e481d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e477fe640 1 -- 172.21.15.6:0/2525501659 reap_dead start 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 shutdown_connections 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 >> 172.21.15.6:0/2525501659 conn(0x7f7e480a2e30 msgr2=0x7f7e4806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 shutdown_connections 2024-04-25T21:40:56.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.675+0000 7f7e4d485640 1 -- 172.21.15.6:0/2525501659 wait complete. 2024-04-25T21:40:56.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T21:40:56.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient: get_monmap_and_config 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient: monmap: 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:56.799960+0000 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:56.799960+0000 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:56.799 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 1 Processor -- start 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 1 -- start start 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient: init 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient(hunting): picked mon.noname-c con 0x7f203c09e860 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient(hunting): picked mon.noname-b con 0x7f203c0a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient(hunting): picked mon.noname-a con 0x7f203c0a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203c09c130 con 0x7f203c0a1680 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203c099520 con 0x7f203c09e860 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203c099b10 con 0x7f203c0a1270 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f2041587640 10 monclient(hunting): authenticate will time out at 1015.709723s 2024-04-25T21:40:56.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f203b7fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f203c0a1680 0x7f203c13d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55816/0 (socket says 172.21.15.6:55816) 2024-04-25T21:40:56.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f203a7fc640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f203c0a1270 0x7f203c139b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41032/0 (socket says 172.21.15.6:41032) 2024-04-25T21:40:56.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.799+0000 7f203b7fe640 1 -- 172.21.15.6:0/244108788 learned_addr learned my addr 172.21.15.6:0/244108788 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f203b7fe640 1 -- 172.21.15.6:0/244108788 >> v1:172.21.15.6:6789/0 conn(0x7f203c0a1680 legacy=0x7f203c13d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f203a7fc640 1 -- 172.21.15.6:0/244108788 >> v1:172.21.15.6:6790/0 conn(0x7f203c0a1270 legacy=0x7f203c139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642519860 0 0) 0x7f203c099b10 con 0x7f203c0a1270 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): my global_id is 4586 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2028003530 con 0x7f203c0a1270 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f203affd640 1 -- 172.21.15.6:0/244108788 >> v1:172.21.15.203:6789/0 conn(0x7f203c09e860 legacy=0x7f203c0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995071662 0 0) 0x7f20340017e0 con 0x7f203c0a1680 2024-04-25T21:40:56.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient(hunting): my global_id is 4620 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f203c099b10 con 0x7f203c0a1680 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2964800365 0 0) 0x7f2028003530 con 0x7f203c0a1270 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 >> v1:172.21.15.203:6789/0 conn(0x7f203c09e860 legacy=0x7f203c0a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 >> v1:172.21.15.6:6789/0 conn(0x7f203c0a1680 legacy=0x7f203c13d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 monclient: found mon.noname-b 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203c09a0d0 con 0x7f203c0a1270 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f20240018d0 con 0x7f203c0a1270 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 5 monclient: authenticate success, global_id 4586 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: dump: 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.806 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2024001c20 con 0x7f203c0a1270 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 4 set_mon_vals no callback set 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 1 -- 172.21.15.6:0/244108788 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2024004990 con 0x7f203c0a1270 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2039ffb640 10 monclient: dump: 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.807 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient: shutdown 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- 172.21.15.6:0/244108788 >> v1:172.21.15.6:6790/0 conn(0x7f203c0a1270 legacy=0x7f203c139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- 172.21.15.6:0/244108788 shutdown_connections 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- 172.21.15.6:0/244108788 >> 172.21.15.6:0/244108788 conn(0x7f203c0a1c50 msgr2=0x7f203c0a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- 172.21.15.6:0/244108788 shutdown_connections 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- 172.21.15.6:0/244108788 wait complete. 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient: build_initial_monmap 2024-04-25T21:40:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient: monmap: 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:56.809 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 Processor -- start 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- start start 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient: init 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient(hunting): picked mon.noname-a con 0x7f203c09e860 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient(hunting): picked mon.noname-b con 0x7f203c0a1270 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient(hunting): picked mon.noname-c con 0x7f203c0a1680 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203c08e350 con 0x7f203c09e860 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203c08d590 con 0x7f203c0a1270 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203c072b40 con 0x7f203c0a1680 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:56.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f2041587640 10 monclient(hunting): authenticate will time out at 1015.714765s 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f203b7fe640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f203c0a1680 0x7f203c1d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41044/0 (socket says 172.21.15.6:41044) 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f203affd640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f203c09e860 0x7f203c08c770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55828/0 (socket says 172.21.15.6:55828) 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.803+0000 7f203b7fe640 1 -- 172.21.15.6:0/2033574026 learned_addr learned my addr 172.21.15.6:0/2033574026 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f203b7fe640 1 -- 172.21.15.6:0/2033574026 >> v1:172.21.15.6:6790/0 conn(0x7f203c0a1680 legacy=0x7f203c1d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f203affd640 1 -- 172.21.15.6:0/2033574026 >> v1:172.21.15.6:6789/0 conn(0x7f203c09e860 legacy=0x7f203c08c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 488793430 0 0) 0x7f203c08e350 con 0x7f203c09e860 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): my global_id is 4623 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f201c003530 con 0x7f203c09e860 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2639091552 0 0) 0x7f203c072b40 con 0x7f203c0a1680 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): my global_id is 4589 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f203c08e350 con 0x7f203c0a1680 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f203a7fc640 1 -- 172.21.15.6:0/2033574026 >> v1:172.21.15.203:6789/0 conn(0x7f203c0a1270 legacy=0x7f203c08ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2712166659 0 0) 0x7f203c08e350 con 0x7f203c0a1680 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f203c072b40 con 0x7f203c0a1680 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f2034001750 con 0x7f203c0a1680 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): dump: 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.815 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2461817977 0 0) 0x7f201c003530 con 0x7f203c09e860 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f203c08e350 con 0x7f203c09e860 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f202c001b00 con 0x7f203c09e860 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): dump: 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.816 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779560095 0 0) 0x7f203c08d590 con 0x7f203c0a1270 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _negotiate 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient(hunting): my global_id is 94222 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f201c003530 con 0x7f203c0a1270 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3034237403 0 0) 0x7f203c072b40 con 0x7f203c0a1680 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 >> v1:172.21.15.203:6789/0 conn(0x7f203c0a1270 legacy=0x7f203c08ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 >> v1:172.21.15.6:6789/0 conn(0x7f203c09e860 legacy=0x7f203c08c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: _finish_hunting 0 2024-04-25T21:40:56.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 monclient: found mon.c 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203c08f8c0 con 0x7f203c0a1680 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: _finish_auth 0 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: _check_auth_tickets 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 5 monclient: authenticate success, global_id 4589 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 10 monclient: _renew_subs 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f203c0903c0 con 0x7f203c0a1680 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 10 monclient: _renew_subs 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2034004f50 con 0x7f203c0a1680 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f203c090a60 con 0x7f203c0a1680 2024-04-25T21:40:56.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f20340052a0 con 0x7f203c0a1680 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 10 monclient: dump: 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:56.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20397fa640 4 set_mon_vals no callback set 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f2034010040 con 0x7f203c0a1680 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f2034044730 con 0x7f203c0a1680 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f203c08ea90 con 0x7f203c0a1680 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.807+0000 7f203affd640 10 monclient: get_auth_request con 0x7f201c036bb0 auth_method 0 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.811+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f203c08ea90 con 0x7f203c0a1680 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.811+0000 7f20337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:56.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.811+0000 7f20337fe640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:56.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.967+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f203c09a3b0 con 0x7f201c036bb0 2024-04-25T21:40:56.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f20337fe640 1 -- 172.21.15.6:0/2033574026 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7f203c09a3b0 con 0x7f201c036bb0 2024-04-25T21:40:56.971 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 >> v1:172.21.15.203:6812/3164820986 conn(0x7f201c036bb0 legacy=0x7f201c039070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 10 monclient: shutdown 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 >> v1:172.21.15.6:6790/0 conn(0x7f203c0a1680 legacy=0x7f203c1d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 shutdown_connections 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 >> 172.21.15.6:0/2033574026 conn(0x7f203c0a1c50 msgr2=0x7f203c098780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 shutdown_connections 2024-04-25T21:40:56.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:56.971+0000 7f2041587640 1 -- 172.21.15.6:0/2033574026 wait complete. 2024-04-25T21:40:56.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:40:57.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient: get_monmap_and_config 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient: build_initial_monmap 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient: monmap: 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:40:57.108883+0000 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:40:57.108883+0000 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:40:57.108 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 1 Processor -- start 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 1 -- start start 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient: init 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient(hunting): picked mon.noname-b con 0x7f6f380a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient(hunting): picked mon.noname-a con 0x7f6f380a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient(hunting): picked mon.noname-c con 0x7f6f3813d680 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f3809adf0 con 0x7f6f380a4970 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f3809b340 con 0x7f6f3813d680 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f3809b620 con 0x7f6f380a3e80 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f3e404640 10 monclient(hunting): authenticate will time out at 1016.018720s 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f377fe640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6f380a4970 0x7f6f3813bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55832/0 (socket says 172.21.15.6:55832) 2024-04-25T21:40:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f37fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6f380a3e80 0x7f6f380a4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41054/0 (socket says 172.21.15.6:41054) 2024-04-25T21:40:57.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.107+0000 7f6f377fe640 1 -- 172.21.15.6:0/128000382 learned_addr learned my addr 172.21.15.6:0/128000382 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f377fe640 1 -- 172.21.15.6:0/128000382 >> v1:172.21.15.6:6789/0 conn(0x7f6f380a4970 legacy=0x7f6f3813bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f37fff640 1 -- 172.21.15.6:0/128000382 >> v1:172.21.15.6:6790/0 conn(0x7f6f380a3e80 legacy=0x7f6f380a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340595854 0 0) 0x7f6f3809adf0 con 0x7f6f380a4970 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): my global_id is 4626 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f24003530 con 0x7f6f380a4970 2024-04-25T21:40:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3c9a1640 1 -- 172.21.15.6:0/128000382 >> v1:172.21.15.203:6789/0 conn(0x7f6f3813d680 legacy=0x7f6f3813fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1852515314 0 0) 0x7f6f3809b620 con 0x7f6f380a3e80 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): _negotiate 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient(hunting): my global_id is 4592 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f3809adf0 con 0x7f6f380a3e80 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3111437162 0 0) 0x7f6f24003530 con 0x7f6f380a4970 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 >> v1:172.21.15.6:6790/0 conn(0x7f6f380a3e80 legacy=0x7f6f380a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 >> v1:172.21.15.203:6789/0 conn(0x7f6f3813d680 legacy=0x7f6f3813fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: _finish_hunting 0 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 monclient: found mon.noname-a 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3806eaf0 con 0x7f6f380a4970 2024-04-25T21:40:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: _finish_auth 0 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: _check_auth_tickets 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 5 monclient: authenticate success, global_id 4626 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6f200018d0 con 0x7f6f380a4970 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: dump: 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:57.115 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f20001c20 con 0x7f6f380a4970 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 1 -- 172.21.15.6:0/128000382 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6f20004a50 con 0x7f6f380a4970 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient: get_monmap_and_config success 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 4 set_mon_vals no callback set 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f36ffd640 10 monclient: dump: 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient: shutdown 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- 172.21.15.6:0/128000382 >> v1:172.21.15.6:6789/0 conn(0x7f6f380a4970 legacy=0x7f6f3813bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- 172.21.15.6:0/128000382 shutdown_connections 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- 172.21.15.6:0/128000382 >> 172.21.15.6:0/128000382 conn(0x7f6f380a5f80 msgr2=0x7f6f380a6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- 172.21.15.6:0/128000382 shutdown_connections 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- 172.21.15.6:0/128000382 wait complete. 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient: build_initial_monmap 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient: monmap: 2024-04-25T21:40:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:40:57.118 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:57.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 Processor -- start 2024-04-25T21:40:57.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- start start 2024-04-25T21:40:57.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient: init 2024-04-25T21:40:57.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient: _reopen_session rank -1 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient(hunting): picked mon.noname-b con 0x7f6f380a3e80 addr v1:172.21.15.203:6789/0 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient(hunting): picked mon.noname-a con 0x7f6f380a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient(hunting): picked mon.noname-c con 0x7f6f3813d680 addr v1:172.21.15.6:6790/0 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f381cc880 con 0x7f6f380a4970 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f380a5c00 con 0x7f6f380a3e80 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f381222c0 con 0x7f6f3813d680 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient(hunting): _renew_subs 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.111+0000 7f6f3e404640 10 monclient(hunting): authenticate will time out at 1016.023900s 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3c9a1640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f6f3813d680 0x7f6f381d43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:41058/0 (socket says 172.21.15.6:41058) 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3c9a1640 1 -- 172.21.15.6:0/175555573 learned_addr learned my addr 172.21.15.6:0/175555573 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f377fe640 1 -- 172.21.15.6:0/175555573 >> v1:172.21.15.6:6789/0 conn(0x7f6f380a4970 legacy=0x7f6f381d0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3c9a1640 1 -- 172.21.15.6:0/175555573 >> v1:172.21.15.6:6790/0 conn(0x7f6f3813d680 legacy=0x7f6f381d43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:57.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3897111813 0 0) 0x7f6f381222c0 con 0x7f6f3813d680 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): my global_id is 4595 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f37fff640 1 -- 172.21.15.6:0/175555573 >> v1:172.21.15.203:6789/0 conn(0x7f6f380a3e80 legacy=0x7f6f38138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f18003530 con 0x7f6f3813d680 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3214873196 0 0) 0x7f6f3806eaf0 con 0x7f6f380a4970 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): my global_id is 4629 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f381222c0 con 0x7f6f380a4970 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1160888466 0 0) 0x7f6f18003530 con 0x7f6f3813d680 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f3806eaf0 con 0x7f6f3813d680 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6f30001ba0 con 0x7f6f3813d680 2024-04-25T21:40:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403444679 0 0) 0x7f6f380a5c00 con 0x7f6f380a3e80 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _negotiate 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:40:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): my global_id is 94228 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f18003530 con 0x7f6f380a3e80 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 31970242 0 0) 0x7f6f381222c0 con 0x7f6f380a4970 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f380a5c00 con 0x7f6f380a4970 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6f20004990 con 0x7f6f380a4970 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient(hunting): dump: 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:57.125 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1688641832 0 0) 0x7f6f3806eaf0 con 0x7f6f3813d680 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 >> v1:172.21.15.203:6789/0 conn(0x7f6f380a3e80 legacy=0x7f6f38138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 >> v1:172.21.15.6:6789/0 conn(0x7f6f380a4970 legacy=0x7f6f381d0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: _finish_hunting 0 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 monclient: found mon.c 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f381245b0 con 0x7f6f3813d680 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: _finish_auth 0 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: _check_auth_tickets 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 5 monclient: authenticate success, global_id 4595 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 10 monclient: _renew_subs 2024-04-25T21:40:57.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f38123d10 con 0x7f6f3813d680 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 10 monclient: _renew_subs 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f38139bb0 con 0x7f6f3813d680 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f300038d0 con 0x7f6f3813d680 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: handle_config config(0 keys) 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f6f30003c20 con 0x7f6f3813d680 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f367fc640 4 set_mon_vals no callback set 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 10 monclient: dump: 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:40:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f6f30003dc0 con 0x7f6f3813d680 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.115+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f1c005090 con 0x7f6f3813d680 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.119+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f6f300442f0 con 0x7f6f3813d680 2024-04-25T21:40:57.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.119+0000 7f6f37fff640 10 monclient: get_auth_request con 0x7f6f18036c90 auth_method 0 2024-04-25T21:40:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.123+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f6f300156b0 con 0x7f6f3813d680 2024-04-25T21:40:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.123+0000 7f6f34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:40:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.123+0000 7f6f34ff9640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:40:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.271+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6f1c000ba0 con 0x7f6f18036c90 2024-04-25T21:40:57.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.275+0000 7f6f34ff9640 1 -- 172.21.15.6:0/175555573 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33735 (unknown 2965378022 0 458160273) 0x7f6f1c000ba0 con 0x7f6f18036c90 2024-04-25T21:40:57.277 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:40:57.278 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:40:57.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 >> v1:172.21.15.203:6812/3164820986 conn(0x7f6f18036c90 legacy=0x7f6f18039150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:57.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 10 monclient: shutdown 2024-04-25T21:40:57.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:40:57.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 >> v1:172.21.15.6:6790/0 conn(0x7f6f3813d680 legacy=0x7f6f381d43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:40:57.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 shutdown_connections 2024-04-25T21:40:57.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 >> 172.21.15.6:0/175555573 conn(0x7f6f380a5f80 msgr2=0x7f6f380a25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:40:57.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 shutdown_connections 2024-04-25T21:40:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:40:57.279+0000 7f6f3e404640 1 -- 172.21.15.6:0/175555573 wait complete. 2024-04-25T21:40:57.294 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-04-25T21:40:55.531254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168632,"kb_used_data":2336,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566062408,"statfs":{"total":579820584960,"available":579647905792,"internally_reserved":0,"allocated":2392064,"data_stored":1513010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003222"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.615368+0000","last_change":"2024-04-25T21:40:07.782045+0000","last_active":"2024-04-25T21:40:35.615368+0000","last_peered":"2024-04-25T21:40:35.615368+0000","last_clean":"2024-04-25T21:40:35.615368+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:35.615368+0000","last_undegraded":"2024-04-25T21:40:35.615368+0000","last_fullsized":"2024-04-25T21:40:35.615368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:03:18.665993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.612967+0000","last_change":"2024-04-25T21:40:07.128410+0000","last_active":"2024-04-25T21:40:35.612967+0000","last_peered":"2024-04-25T21:40:35.612967+0000","last_clean":"2024-04-25T21:40:35.612967+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:35.612967+0000","last_undegraded":"2024-04-25T21:40:35.612967+0000","last_fullsized":"2024-04-25T21:40:35.612967+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:07:06.819511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.616412+0000","last_change":"2024-04-25T21:40:07.123449+0000","last_active":"2024-04-25T21:40:35.616412+0000","last_peered":"2024-04-25T21:40:35.616412+0000","last_clean":"2024-04-25T21:40:35.616412+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:35.616412+0000","last_undegraded":"2024-04-25T21:40:35.616412+0000","last_fullsized":"2024-04-25T21:40:35.616412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:37:10.700186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614816+0000","last_change":"2024-04-25T21:40:07.129045+0000","last_active":"2024-04-25T21:40:35.614816+0000","last_peered":"2024-04-25T21:40:35.614816+0000","last_clean":"2024-04-25T21:40:35.614816+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:35.614816+0000","last_undegraded":"2024-04-25T21:40:35.614816+0000","last_fullsized":"2024-04-25T21:40:35.614816+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:44:29.487858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.617201+0000","last_change":"2024-04-25T21:40:07.134356+0000","last_active":"2024-04-25T21:40:35.617201+0000","last_peered":"2024-04-25T21:40:35.617201+0000","last_clean":"2024-04-25T21:40:35.617201+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:35.617201+0000","last_undegraded":"2024-04-25T21:40:35.617201+0000","last_fullsized":"2024-04-25T21:40:35.617201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:40:30.010221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:34.534127+0000","last_change":"2024-04-25T21:40:07.126120+0000","last_active":"2024-04-25T21:40:34.534127+0000","last_peered":"2024-04-25T21:40:34.534127+0000","last_clean":"2024-04-25T21:40:34.534127+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:34.534127+0000","last_undegraded":"2024-04-25T21:40:34.534127+0000","last_fullsized":"2024-04-25T21:40:34.534127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:39:34.718302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613555+0000","last_change":"2024-04-25T21:40:07.126420+0000","last_active":"2024-04-25T21:40:35.613555+0000","last_peered":"2024-04-25T21:40:35.613555+0000","last_clean":"2024-04-25T21:40:35.613555+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:35.613555+0000","last_undegraded":"2024-04-25T21:40:35.613555+0000","last_fullsized":"2024-04-25T21:40:35.613555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:51:34.692071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.614625+0000","last_change":"2024-04-25T21:40:07.134996+0000","last_active":"2024-04-25T21:40:35.614625+0000","last_peered":"2024-04-25T21:40:35.614625+0000","last_clean":"2024-04-25T21:40:35.614625+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:35.614625+0000","last_undegraded":"2024-04-25T21:40:35.614625+0000","last_fullsized":"2024-04-25T21:40:35.614625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:05.449585+0000","last_clean_scrub_stamp":"2024-04-25T21:40:05.449585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:39.747689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":179,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:35.613838+0000","last_change":"2024-04-25T21:40:01.342233+0000","last_active":"2024-04-25T21:40:35.613838+0000","last_peered":"2024-04-25T21:40:35.613838+0000","last_clean":"2024-04-25T21:40:35.613838+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:35.613838+0000","last_undegraded":"2024-04-25T21:40:35.613838+0000","last_fullsized":"2024-04-25T21:40:35.613838+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:00.320225+0000","last_clean_scrub_stamp":"2024-04-25T21:40:00.320225+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:45:40.365284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607565,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640269,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:40:57.297 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.297 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.297 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.297 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.297 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.297 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.297 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.298 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T21:40:05.449585+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=5, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.298 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T21:40:00.320225+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=0, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=21, tm_min=40, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T21:40:57.298 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T21:41:17.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.430+0000 7f1b4e53e640 10 monclient: get_monmap_and_config 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.430+0000 7f1b4e53e640 10 monclient: build_initial_monmap 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.430+0000 7f1b4e53e640 10 monclient: monmap: 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:41:17.434456+0000 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:41:17.434456+0000 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:41:17.434 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:41:17.435 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.430+0000 7f1b4e53e640 1 Processor -- start 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.430+0000 7f1b4e53e640 1 -- start start 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.430+0000 7f1b4e53e640 10 monclient: init 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient: _reopen_session rank -1 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b4809e860 addr v1:172.21.15.6:6790/0 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b480a1270 addr v1:172.21.15.203:6789/0 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b480a1680 addr v1:172.21.15.6:6789/0 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4809c130 con 0x7f1b480a1680 2024-04-25T21:41:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b48099520 con 0x7f1b480a1270 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b48099b10 con 0x7f1b4809e860 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient(hunting): _renew_subs 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient(hunting): authenticate will time out at 1036.344559s 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b47fff640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7f1b4809e860 0x7f1b480a0b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47530/0 (socket says 172.21.15.6:47530) 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b47fff640 1 -- 172.21.15.6:0/2751357433 learned_addr learned my addr 172.21.15.6:0/2751357433 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b47fff640 1 -- 172.21.15.6:0/2751357433 >> v1:172.21.15.6:6790/0 conn(0x7f1b4809e860 legacy=0x7f1b480a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b47fff640 1 -- 172.21.15.6:0/2751357433 >> v1:172.21.15.6:6790/0 conn(0x7f1b4809e860 legacy=0x7f1b480a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T21:41:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b47fff640 1 --1- 172.21.15.6:0/2751357433 >> v1:172.21.15.6:6790/0 conn(0x7f1b4809e860 0x7f1b480a0b60 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-04-25T21:41:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4cadb640 1 -- 172.21.15.6:0/2751357433 >> v1:172.21.15.6:6789/0 conn(0x7f1b480a1680 legacy=0x7f1b4813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1575387622 0 0) 0x7f1b40001830 con 0x7f1b480a1680 2024-04-25T21:41:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient(hunting): _negotiate 2024-04-25T21:41:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient(hunting): my global_id is 4632 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b34003530 con 0x7f1b480a1680 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b3ffff640 1 -- 172.21.15.6:0/2751357433 >> v1:172.21.15.203:6789/0 conn(0x7f1b480a1270 legacy=0x7f1b48139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1011155996 0 0) 0x7f1b34003530 con 0x7f1b480a1680 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 >> v1:172.21.15.6:6790/0 conn(0x7f1b4809e860 legacy=0x7f1b480a0b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 >> v1:172.21.15.203:6789/0 conn(0x7f1b480a1270 legacy=0x7f1b48139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: _finish_hunting 0 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 monclient: found mon.noname-a 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.6:6789/0 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4809a0d0 con 0x7f1b480a1680 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: _finish_auth 0 2024-04-25T21:41:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: _check_auth_tickets 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f1b40001f30 con 0x7f1b480a1680 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: dump: 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.441 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 5 monclient: authenticate success, global_id 4632 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 <== mon.0 v1:172.21.15.6:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1b400039a0 con 0x7f1b480a1680 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 10 monclient: get_monmap_and_config success 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b4e53e640 4 set_mon_vals no callback set 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 1 -- 172.21.15.6:0/2751357433 <== mon.0 v1:172.21.15.6:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f1b40003d70 con 0x7f1b480a1680 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.434+0000 7f1b477fe640 10 monclient: dump: 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient: shutdown 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- 172.21.15.6:0/2751357433 >> v1:172.21.15.6:6789/0 conn(0x7f1b480a1680 legacy=0x7f1b4813d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- 172.21.15.6:0/2751357433 shutdown_connections 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- 172.21.15.6:0/2751357433 >> 172.21.15.6:0/2751357433 conn(0x7f1b480a1c50 msgr2=0x7f1b480a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- 172.21.15.6:0/2751357433 shutdown_connections 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- 172.21.15.6:0/2751357433 wait complete. 2024-04-25T21:41:17.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient: build_initial_monmap 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient: monmap: 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:41:17.444 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 Processor -- start 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- start start 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient: init 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient: _reopen_session rank -1 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b4809e860 addr v1:172.21.15.6:6789/0 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b480a1270 addr v1:172.21.15.6:6790/0 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b480a1680 addr v1:172.21.15.203:6789/0 2024-04-25T21:41:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4808d3d0 con 0x7f1b4809e860 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b48072b40 con 0x7f1b480a1680 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4808f780 con 0x7f1b480a1270 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient(hunting): _renew_subs 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b4e53e640 10 monclient(hunting): authenticate will time out at 1036.349859s 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b47fff640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1b4809e860 0x7f1b4808c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54138/0 (socket says 172.21.15.6:54138) 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.438+0000 7f1b47fff640 1 -- 172.21.15.6:0/695797592 learned_addr learned my addr 172.21.15.6:0/695797592 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b3ffff640 1 -- 172.21.15.6:0/695797592 >> v1:172.21.15.6:6790/0 conn(0x7f1b480a1270 legacy=0x7f1b4808ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b47fff640 1 -- 172.21.15.6:0/695797592 >> v1:172.21.15.6:6789/0 conn(0x7f1b4809e860 legacy=0x7f1b4808c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2637273304 0 0) 0x7f1b4808d3d0 con 0x7f1b4809e860 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _negotiate 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): my global_id is 4635 2024-04-25T21:41:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b28003530 con 0x7f1b4809e860 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3266884191 0 0) 0x7f1b4808f780 con 0x7f1b480a1270 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _negotiate 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): my global_id is 4598 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b4808d3d0 con 0x7f1b480a1270 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4cadb640 1 -- 172.21.15.6:0/695797592 >> v1:172.21.15.203:6789/0 conn(0x7f1b480a1680 legacy=0x7f1b481d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 726714962 0 0) 0x7f1b28003530 con 0x7f1b4809e860 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b47fff640 1 --1- 172.21.15.6:0/695797592 >> v1:172.21.15.6:6789/0 conn(0x7f1b4809e860 0x7f1b4808c5b0 :-1 s=OPENED pgs=245 cs=1 l=1).queue_received will delay after 1.71408e+09 on 0x7f1b380014e0 mon_map magic: 0 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b4808f780 con 0x7f1b4809e860 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 678806479 0 0) 0x7f1b4808d3d0 con 0x7f1b480a1270 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b28003530 con 0x7f1b480a1270 2024-04-25T21:41:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f1b30001b40 con 0x7f1b480a1270 2024-04-25T21:41:17.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:41:17.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-04-25T21:41:17.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): dump: 2024-04-25T21:41:17.451 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.451 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.451 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.451 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f1b380014e0 con 0x7f1b4809e860 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): dump: 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.452 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578155107 0 0) 0x7f1b48072b40 con 0x7f1b480a1680 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _negotiate 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient(hunting): my global_id is 94231 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b4808d3d0 con 0x7f1b480a1680 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 420320135 0 0) 0x7f1b28003530 con 0x7f1b480a1270 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 >> v1:172.21.15.203:6789/0 conn(0x7f1b480a1680 legacy=0x7f1b481d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 >> v1:172.21.15.6:6789/0 conn(0x7f1b4809e860 legacy=0x7f1b4808c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: _finish_hunting 0 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 monclient: found mon.c 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4808e1d0 con 0x7f1b480a1270 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: _finish_auth 0 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: _check_auth_tickets 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1341081284 0 0) 0x7f1b4808f780 con 0x7f1b4809e860 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 5 monclient: authenticate success, global_id 4598 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 10 monclient: _renew_subs 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:41:17.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b48090240 con 0x7f1b480a1270 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 10 monclient: _renew_subs 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b480908e0 con 0x7f1b480a1270 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1b30003950 con 0x7f1b480a1270 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7f1b30003ca0 con 0x7f1b480a1270 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b46ffd640 4 set_mon_vals no callback set 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 10 monclient: dump: 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.455 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.6:6790/0 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.6:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b2c005090 con 0x7f1b480a1270 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7f1b30003e40 con 0x7f1b480a1270 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7f1b30044310 con 0x7f1b480a1270 2024-04-25T21:41:17.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.442+0000 7f1b47fff640 10 monclient: get_auth_request con 0x7f1b28036cb0 auth_method 0 2024-04-25T21:41:17.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.446+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mon.2 v1:172.21.15.6:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7f1b48090240 con 0x7f1b480a1270 2024-04-25T21:41:17.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.446+0000 7f1b457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:41:17.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.446+0000 7f1b457fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:41:17.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.606+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 --> v1:172.21.15.203:6812/3164820986 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b2c000ba0 con 0x7f1b28036cb0 2024-04-25T21:41:17.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b457fa640 1 -- 172.21.15.6:0/695797592 <== mgr.4101 v1:172.21.15.203:6812/3164820986 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33741 (unknown 2965378022 0 1141458210) 0x7f1b2c000ba0 con 0x7f1b28036cb0 2024-04-25T21:41:17.612 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-04-25T21:41:17.612 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-04-25T21:41:17.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 >> v1:172.21.15.203:6812/3164820986 conn(0x7f1b28036cb0 legacy=0x7f1b28039170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 10 monclient: shutdown 2024-04-25T21:41:17.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:41:17.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 >> v1:172.21.15.6:6790/0 conn(0x7f1b480a1270 legacy=0x7f1b4808ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 shutdown_connections 2024-04-25T21:41:17.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 >> 172.21.15.6:0/695797592 conn(0x7f1b480a1c50 msgr2=0x7f1b48098ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:41:17.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 shutdown_connections 2024-04-25T21:41:17.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.610+0000 7f1b4e53e640 1 -- 172.21.15.6:0/695797592 wait complete. 2024-04-25T21:41:17.631 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-04-25T21:41:17.537189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168600,"kb_used_data":2336,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566062440,"statfs":{"total":579820584960,"available":579647938560,"internally_reserved":0,"allocated":2392064,"data_stored":1513010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003226"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:59.593695+0000","last_change":"2024-04-25T21:40:59.593746+0000","last_active":"2024-04-25T21:40:59.593695+0000","last_peered":"2024-04-25T21:40:59.593695+0000","last_clean":"2024-04-25T21:40:59.593695+0000","last_became_active":"2024-04-25T21:40:07.781399+0000","last_became_peered":"2024-04-25T21:40:07.781399+0000","last_unstale":"2024-04-25T21:40:59.593695+0000","last_undegraded":"2024-04-25T21:40:59.593695+0000","last_fullsized":"2024-04-25T21:40:59.593695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:59.593519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:59.593519+0000","last_clean_scrub_stamp":"2024-04-25T21:40:59.593519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:16:40.316774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:55.617122+0000","last_change":"2024-04-25T21:40:55.617141+0000","last_active":"2024-04-25T21:40:55.617122+0000","last_peered":"2024-04-25T21:40:55.617122+0000","last_clean":"2024-04-25T21:40:55.617122+0000","last_became_active":"2024-04-25T21:40:07.127678+0000","last_became_peered":"2024-04-25T21:40:07.127678+0000","last_unstale":"2024-04-25T21:40:55.617122+0000","last_undegraded":"2024-04-25T21:40:55.617122+0000","last_fullsized":"2024-04-25T21:40:55.617122+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:55.617046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:55.617046+0000","last_clean_scrub_stamp":"2024-04-25T21:40:55.617046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:03:06.579201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:57.654186+0000","last_change":"2024-04-25T21:40:57.654240+0000","last_active":"2024-04-25T21:40:57.654186+0000","last_peered":"2024-04-25T21:40:57.654186+0000","last_clean":"2024-04-25T21:40:57.654186+0000","last_became_active":"2024-04-25T21:40:07.122823+0000","last_became_peered":"2024-04-25T21:40:07.122823+0000","last_unstale":"2024-04-25T21:40:57.654186+0000","last_undegraded":"2024-04-25T21:40:57.654186+0000","last_fullsized":"2024-04-25T21:40:57.654186+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:57.654007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:57.654007+0000","last_clean_scrub_stamp":"2024-04-25T21:40:57.654007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:08:43.613021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:56.612253+0000","last_change":"2024-04-25T21:40:56.612382+0000","last_active":"2024-04-25T21:40:56.612253+0000","last_peered":"2024-04-25T21:40:56.612253+0000","last_clean":"2024-04-25T21:40:56.612253+0000","last_became_active":"2024-04-25T21:40:07.126539+0000","last_became_peered":"2024-04-25T21:40:07.126539+0000","last_unstale":"2024-04-25T21:40:56.612253+0000","last_undegraded":"2024-04-25T21:40:56.612253+0000","last_fullsized":"2024-04-25T21:40:56.612253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:56.612051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:56.612051+0000","last_clean_scrub_stamp":"2024-04-25T21:40:56.612051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:06.630307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:57.731295+0000","last_change":"2024-04-25T21:40:57.731350+0000","last_active":"2024-04-25T21:40:57.731295+0000","last_peered":"2024-04-25T21:40:57.731295+0000","last_clean":"2024-04-25T21:40:57.731295+0000","last_became_active":"2024-04-25T21:40:07.133623+0000","last_became_peered":"2024-04-25T21:40:07.133623+0000","last_unstale":"2024-04-25T21:40:57.731295+0000","last_undegraded":"2024-04-25T21:40:57.731295+0000","last_fullsized":"2024-04-25T21:40:57.731295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:57.731114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:57.731114+0000","last_clean_scrub_stamp":"2024-04-25T21:40:57.731114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:14:32.575688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:56.222285+0000","last_change":"2024-04-25T21:40:56.222325+0000","last_active":"2024-04-25T21:40:56.222285+0000","last_peered":"2024-04-25T21:40:56.222285+0000","last_clean":"2024-04-25T21:40:56.222285+0000","last_became_active":"2024-04-25T21:40:07.125632+0000","last_became_peered":"2024-04-25T21:40:07.125632+0000","last_unstale":"2024-04-25T21:40:56.222285+0000","last_undegraded":"2024-04-25T21:40:56.222285+0000","last_fullsized":"2024-04-25T21:40:56.222285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:56.222188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:56.222188+0000","last_clean_scrub_stamp":"2024-04-25T21:40:56.222188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:28.016317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:58.626494+0000","last_change":"2024-04-25T21:40:58.626624+0000","last_active":"2024-04-25T21:40:58.626494+0000","last_peered":"2024-04-25T21:40:58.626494+0000","last_clean":"2024-04-25T21:40:58.626494+0000","last_became_active":"2024-04-25T21:40:07.125621+0000","last_became_peered":"2024-04-25T21:40:07.125621+0000","last_unstale":"2024-04-25T21:40:58.626494+0000","last_undegraded":"2024-04-25T21:40:58.626494+0000","last_fullsized":"2024-04-25T21:40:58.626494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:58.626287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:58.626287+0000","last_clean_scrub_stamp":"2024-04-25T21:40:58.626287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:27:37.403145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:58.730878+0000","last_change":"2024-04-25T21:40:58.730934+0000","last_active":"2024-04-25T21:40:58.730878+0000","last_peered":"2024-04-25T21:40:58.730878+0000","last_clean":"2024-04-25T21:40:58.730878+0000","last_became_active":"2024-04-25T21:40:07.134269+0000","last_became_peered":"2024-04-25T21:40:07.134269+0000","last_unstale":"2024-04-25T21:40:58.730878+0000","last_undegraded":"2024-04-25T21:40:58.730878+0000","last_fullsized":"2024-04-25T21:40:58.730878+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T21:40:58.730707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T21:40:58.730707+0000","last_clean_scrub_stamp":"2024-04-25T21:40:58.730707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:30:28.952151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'119","reported_seq":190,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-25T21:40:56.651583+0000","last_change":"2024-04-25T21:40:56.651655+0000","last_active":"2024-04-25T21:40:56.651583+0000","last_peered":"2024-04-25T21:40:56.651583+0000","last_clean":"2024-04-25T21:40:56.651583+0000","last_became_active":"2024-04-25T21:40:01.341384+0000","last_became_peered":"2024-04-25T21:40:01.341384+0000","last_unstale":"2024-04-25T21:40:56.651583+0000","last_undegraded":"2024-04-25T21:40:56.651583+0000","last_fullsized":"2024-04-25T21:40:56.651583+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'119","last_scrub_stamp":"2024-04-25T21:40:56.651410+0000","last_deep_scrub":"25'119","last_deep_scrub_stamp":"2024-04-25T21:40:56.651410+0000","last_clean_scrub_stamp":"2024-04-25T21:40:56.651410+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:16:21.983273+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":12,"seq":51539607569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607569,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607570,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31912,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339928,"statfs":{"total":96636764160,"available":96604086272,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640273,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":753664,"data_stored":604787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T21:41:17.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T21:41:17.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.754+0000 7fa16ed9d640 10 monclient: get_monmap_and_config 2024-04-25T21:41:17.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.754+0000 7fa16ed9d640 10 monclient: build_initial_monmap 2024-04-25T21:41:17.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.754+0000 7fa16ed9d640 10 monclient: monmap: 2024-04-25T21:41:17.758 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:41:17.758432+0000 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:41:17.758432+0000 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-b 2024-04-25T21:41:17.759 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.754+0000 7fa16ed9d640 1 Processor -- start 2024-04-25T21:41:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.754+0000 7fa16ed9d640 1 -- start start 2024-04-25T21:41:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.754+0000 7fa16ed9d640 10 monclient: init 2024-04-25T21:41:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient: _reopen_session rank -1 2024-04-25T21:41:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-25T21:41:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa1680a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:41:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa1680a4970 addr v1:172.21.15.6:6789/0 2024-04-25T21:41:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa16813d5a0 addr v1:172.21.15.203:6789/0 2024-04-25T21:41:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16809adf0 con 0x7fa1680a4970 2024-04-25T21:41:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16809b340 con 0x7fa16813d5a0 2024-04-25T21:41:17.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16809b620 con 0x7fa1680a3e80 2024-04-25T21:41:17.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient(hunting): _renew_subs 2024-04-25T21:41:17.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient(hunting): authenticate will time out at 1036.668754s 2024-04-25T21:41:17.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16cb39640 1 --1- >> v1:172.21.15.6:6790/0 conn(0x7fa1680a3e80 0x7fa1680a4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6790/0 says I am v1:172.21.15.6:47548/0 (socket says 172.21.15.6:47548) 2024-04-25T21:41:17.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16cb39640 1 -- 172.21.15.6:0/1359647056 learned_addr learned my addr 172.21.15.6:0/1359647056 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:41:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16cb39640 1 -- 172.21.15.6:0/1359647056 >> v1:172.21.15.6:6790/0 conn(0x7fa1680a3e80 legacy=0x7fa1680a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa167fff640 1 -- 172.21.15.6:0/1359647056 >> v1:172.21.15.6:6789/0 conn(0x7fa1680a4970 legacy=0x7fa16813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 <== mon.2 v1:172.21.15.6:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516975404 0 0) 0x7fa16809b620 con 0x7fa1680a3e80 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): _negotiate 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): my global_id is 4601 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 --> v1:172.21.15.6:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa154003530 con 0x7fa1680a3e80 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873028695 0 0) 0x7fa16809adf0 con 0x7fa1680a4970 2024-04-25T21:41:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): _negotiate 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient(hunting): my global_id is 4638 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa16809b620 con 0x7fa1680a4970 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16d33a640 1 -- 172.21.15.6:0/1359647056 >> v1:172.21.15.203:6789/0 conn(0x7fa16813d5a0 legacy=0x7fa16813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 <== mon.2 v1:172.21.15.6:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3560149473 0 0) 0x7fa154003530 con 0x7fa1680a3e80 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 >> v1:172.21.15.203:6789/0 conn(0x7fa16813d5a0 legacy=0x7fa16813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 >> v1:172.21.15.6:6789/0 conn(0x7fa1680a4970 legacy=0x7fa16813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: _finish_hunting 0 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 monclient: found mon.noname-b 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.6:6790/0 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 --> v1:172.21.15.6:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16806eaf0 con 0x7fa1680a3e80 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: _finish_auth 0 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: _check_auth_tickets 2024-04-25T21:41:17.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 <== mon.2 v1:172.21.15.6:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa158001c50 con 0x7fa1680a3e80 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 5 monclient: authenticate success, global_id 4601 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: dump: 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.775 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.776 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 <== mon.2 v1:172.21.15.6:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa158001fa0 con 0x7fa1680a3e80 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: handle_config config(0 keys) 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 1 -- 172.21.15.6:0/1359647056 <== mon.2 v1:172.21.15.6:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa158004dd0 con 0x7fa1680a3e80 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa1677fe640 10 monclient: dump: 2024-04-25T21:41:17.777 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 10 monclient: get_monmap_and_config success 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.758+0000 7fa16ed9d640 4 set_mon_vals no callback set 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient: shutdown 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1359647056 >> v1:172.21.15.6:6790/0 conn(0x7fa1680a3e80 legacy=0x7fa1680a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1359647056 shutdown_connections 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1359647056 >> 172.21.15.6:0/1359647056 conn(0x7fa1680a9e30 msgr2=0x7fa1680a5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1359647056 shutdown_connections 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1359647056 wait complete. 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient: build_initial_monmap 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient: monmap: 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2024-04-25T21:41:17.780 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2024-04-25T21:41:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2024-04-25T21:41:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2024-04-25T21:41:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a 2024-04-25T21:41:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-04-25T21:41:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.noname-c 2024-04-25T21:41:17.781 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 Processor -- start 2024-04-25T21:41:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- start start 2024-04-25T21:41:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient: init 2024-04-25T21:41:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient: _reopen_session rank -1 2024-04-25T21:41:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-25T21:41:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa1680a3e80 addr v1:172.21.15.6:6790/0 2024-04-25T21:41:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa1680a4970 addr v1:172.21.15.203:6789/0 2024-04-25T21:41:17.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa16813d5a0 addr v1:172.21.15.6:6789/0 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16813bea0 con 0x7fa16813d5a0 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa168140e00 con 0x7fa1680a4970 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 1 -- --> v1:172.21.15.6:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16813a450 con 0x7fa1680a3e80 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient(hunting): _renew_subs 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16ed9d640 10 monclient(hunting): authenticate will time out at 1036.673998s 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16d33a640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa16813d5a0 0x7fa1681d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54146/0 (socket says 172.21.15.6:54146) 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.762+0000 7fa16d33a640 1 -- 172.21.15.6:0/1750573833 learned_addr learned my addr 172.21.15.6:0/1750573833 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-04-25T21:41:17.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16d33a640 1 -- 172.21.15.6:0/1750573833 >> v1:172.21.15.6:6789/0 conn(0x7fa16813d5a0 legacy=0x7fa1681d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16cb39640 1 -- 172.21.15.6:0/1750573833 >> v1:172.21.15.6:6790/0 conn(0x7fa1680a3e80 legacy=0x7fa1681cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437245923 0 0) 0x7fa16809b340 con 0x7fa16813d5a0 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): _negotiate 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): _init_auth method 2 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): my global_id is 4641 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa148003530 con 0x7fa16813d5a0 2024-04-25T21:41:17.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa167fff640 1 -- 172.21.15.6:0/1750573833 >> v1:172.21.15.203:6789/0 conn(0x7fa1680a4970 legacy=0x7fa1681d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1344418148 0 0) 0x7fa148003530 con 0x7fa16813d5a0 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 --> v1:172.21.15.6:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa16809b340 con 0x7fa16813d5a0 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa160001ba0 con 0x7fa16813d5a0 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): dump: 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.785 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient(hunting): _finish_auth 0 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1715438904 0 0) 0x7fa16809b340 con 0x7fa16813d5a0 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 >> v1:172.21.15.6:6790/0 conn(0x7fa1680a3e80 legacy=0x7fa1681cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 >> v1:172.21.15.203:6789/0 conn(0x7fa1680a4970 legacy=0x7fa1681d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: _finish_hunting 0 2024-04-25T21:41:17.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 monclient: found mon.a 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16813c740 con 0x7fa16813d5a0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: _finish_auth 0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: _check_auth_tickets 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 5 monclient: authenticate success, global_id 4641 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 10 monclient: _renew_subs 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 --> v1:172.21.15.6:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa16813ad90 con 0x7fa16813d5a0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 10 monclient: _renew_subs 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 --> v1:172.21.15.6:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa16813cd30 con 0x7fa16813d5a0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1600038d0 con 0x7fa16813d5a0 2024-04-25T21:41:17.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: handle_config config(0 keys) 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 1408576389 0 0) 0x7fa160003c20 con 0x7fa16813d5a0 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa166ffd640 4 set_mon_vals no callback set 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 10 monclient: dump: 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 7c01789f-5916-4cc5-8455-2e5d8f8fad2b 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:created 2024-04-25T21:39:14.743126+0000 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 19 (squid) 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 3 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.a 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-04-25T21:41:17.788 INFO:teuthology.orchestra.run.smithi006.stderr:2: v1:172.21.15.6:6790/0 mon.c 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (unknown 1064497996 0 0) 0x7fa160003e00 con 0x7fa16813d5a0 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 --> v1:172.21.15.6:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa14c005090 con 0x7fa16813d5a0 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 8 ==== osd_map(25..25 src has 1..25) ==== 3506+0+0 (unknown 3067585614 0 0) 0x7fa1600443c0 con 0x7fa16813d5a0 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.766+0000 7fa16cb39640 10 monclient: get_auth_request con 0x7fa148036a20 auth_method 0 2024-04-25T21:41:17.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.770+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (unknown 1092875540 0 3292399231) 0x7fa16813ad90 con 0x7fa16813d5a0 2024-04-25T21:41:17.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.770+0000 7fa1657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-25T21:41:17.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.770+0000 7fa1657fa640 10 monclient: _finish_command 1 = system:0 2024-04-25T21:41:17.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.934+0000 7fa16ed9d640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-25T21:41:17.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.934+0000 7fa16ed9d640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-25T21:41:17.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.934+0000 7fa16ed9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.6:6789/0 2024-04-25T21:41:17.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.934+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 --> v1:172.21.15.6:6789/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa14c005380 con 0x7fa16813d5a0 2024-04-25T21:41:17.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.938+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) ==== 123+0+0 (unknown 3414686427 0 0) 0x7fa16813bea0 con 0x7fa16813d5a0 2024-04-25T21:41:17.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.938+0000 7fa1657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-25T21:41:17.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.938+0000 7fa1657fa640 10 monclient: _finish_command 2 = system:0 2024-04-25T21:41:17.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.942+0000 7fa1657fa640 1 -- 172.21.15.6:0/1750573833 <== mon.0 v1:172.21.15.6:6789/0 11 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7fa1600157c0 con 0x7fa16813d5a0 2024-04-25T21:41:17.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.942+0000 7fa1657fa640 10 monclient: handle_config config(1 keys) 2024-04-25T21:41:17.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.942+0000 7fa1667fc640 4 set_mon_vals no callback set 2024-04-25T21:41:17.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.942+0000 7fa1667fc640 10 set_mon_vals mon_health_to_clog = false 2024-04-25T21:41:17.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 >> v1:172.21.15.203:6812/3164820986 conn(0x7fa148036a20 legacy=0x7fa148038ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 10 monclient: shutdown 2024-04-25T21:41:17.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-25T21:41:17.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 >> v1:172.21.15.6:6789/0 conn(0x7fa16813d5a0 legacy=0x7fa1681d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T21:41:17.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 shutdown_connections 2024-04-25T21:41:17.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 >> 172.21.15.6:0/1750573833 conn(0x7fa1680a9e30 msgr2=0x7fa1680a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T21:41:17.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 shutdown_connections 2024-04-25T21:41:17.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-04-25T21:41:17.946+0000 7fa16ed9d640 1 -- 172.21.15.6:0/1750573833 wait complete. 2024-04-25T21:41:17.964 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T21:41:17.964 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T21:41:17.964 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T21:41:17.964 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.046 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T21:41:18.046 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T21:41:18.046 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.146 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T21:41:18.147 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T21:41:18.147 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.235 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T21:41:18.235 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T21:41:18.235 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.309 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T21:41:18.309 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T21:41:18.309 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.401 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T21:41:18.401 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T21:41:18.401 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.486 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T21:41:18.487 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T21:41:18.487 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T21:41:18.487 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.516 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T21:41:18.516 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T21:41:18.517 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T21:41:18.517 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.540 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T21:41:18.540 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T21:41:18.540 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.602 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T21:41:18.602 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T21:41:18.603 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T21:41:18.617 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T21:41:18.617 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T21:41:18.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | head -n 1 2024-04-25T21:41:18.662 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:41:18.662 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T21:41:18.902 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:41:18.902 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T21:41:19.067 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:41:19.067 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T21:41:19.209 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:41:19.209 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T21:41:19.356 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:41:19.356 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T21:41:19.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:41:19.536 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T21:41:19.695 INFO:tasks.ceph:Archiving mon data... 2024-04-25T21:41:19.696 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/data/mon.a.tgz 2024-04-25T21:41:19.696 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-25T21:41:19.702 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.QAQKngCBMk 2024-04-25T21:41:19.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QAQKngCBMk 2024-04-25T21:41:19.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.QAQKngCBMk 2024-04-25T21:41:19.860 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.QAQKngCBMk is 184KB 2024-04-25T21:41:19.876 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.QAQKngCBMk 2024-04-25T21:41:19.881 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/data/mon.c.tgz 2024-04-25T21:41:19.881 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-04-25T21:41:19.927 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.RDrkXhQvAp 2024-04-25T21:41:19.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.RDrkXhQvAp 2024-04-25T21:41:20.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.RDrkXhQvAp 2024-04-25T21:41:20.096 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.RDrkXhQvAp is 185KB 2024-04-25T21:41:20.111 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.RDrkXhQvAp 2024-04-25T21:41:20.116 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/data/mon.b.tgz 2024-04-25T21:41:20.116 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-04-25T21:41:20.122 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.1uw3cRs9jF 2024-04-25T21:41:20.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.1uw3cRs9jF 2024-04-25T21:41:20.213 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.1uw3cRs9jF 2024-04-25T21:41:20.274 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.1uw3cRs9jF is 185KB 2024-04-25T21:41:20.291 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.1uw3cRs9jF 2024-04-25T21:41:20.296 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T21:41:20.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T21:41:20.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T21:41:20.359 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T21:41:20.399 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T21:41:20.399 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T21:41:20.401 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/remote/smithi006/crash 2024-04-25T21:41:20.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T21:41:20.415 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/remote/smithi203/crash 2024-04-25T21:41:20.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T21:41:20.455 INFO:tasks.ceph:Compressing logs... 2024-04-25T21:41:20.455 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T21:41:20.460 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T21:41:20.489 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29705.log 2024-04-25T21:41:20.490 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28971.log 2024-04-25T21:41:20.490 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29523.log 2024-04-25T21:41:20.491 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28971.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29705.log.gz 2024-04-25T21:41:20.491 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T21:41:20.492 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28971.log.gz 2024-04-25T21:41:20.492 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28511.log 2024-04-25T21:41:20.492 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29523.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28523.log 2024-04-25T21:41:20.492 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28511.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29523.log.gz 2024-04-25T21:41:20.492 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28671.log 2024-04-25T21:41:20.496 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28511.log.gz 2024-04-25T21:41:20.496 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28699.log 2024-04-25T21:41:20.496 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29233.log 2024-04-25T21:41:20.496 INFO:teuthology.orchestra.run.smithi006.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28523.log.gz 2024-04-25T21:41:20.496 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28699.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28671.log.gz 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28919.log 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28699.log.gz 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29023.log 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29233.log: /var/log/ceph/ceph-client.admin.28919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28192.log 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29023.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28919.log.gz 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28321.log 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29233.log.gz 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29023.log.gz 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29103.log 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28192.log: /var/log/ceph/ceph-client.admin.28321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29861.log 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28192.log.gz 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29103.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28321.log.gz 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29129.log 2024-04-25T21:41:20.497 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29103.log.gz 2024-04-25T21:41:20.498 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29341.log 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29129.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29861.log.gz 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29129.log.gz 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.29341.log: --verbose -- /var/log/ceph/ceph-client.admin.29051.log 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29051.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29341.log.gz 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28353.log 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29051.log.gz 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28520.log 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28367.log 2024-04-25T21:41:20.501 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28353.log.gz 2024-04-25T21:41:20.502 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28136.log 2024-04-25T21:41:20.502 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28367.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28520.log.gz 2024-04-25T21:41:20.502 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23948.log 2024-04-25T21:41:20.502 INFO:teuthology.orchestra.run.smithi006.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28367.log.gz 2024-04-25T21:41:20.502 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28997.log 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28136.log: /var/log/ceph/ceph-client.admin.23948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29835.log 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.23948.log.gz 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28136.log.gz 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29419.log 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28521.log 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28997.log.gz 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29419.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29835.log.gz 2024-04-25T21:41:20.505 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28217.log 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29419.log.gz 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29181.log 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28670.log 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28217.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28521.log.gz 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28518.log 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28217.log.gz 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28670.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.29181.log.gz 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28670.log.gz 2024-04-25T21:41:20.506 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28841.log 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28663.log/var/log/ceph/ceph-osd.1.log: 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28518.log.gz 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28841.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20123.log 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28663.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28841.log.gz 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28355.log 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28663.log.gz 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.20123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29627.log 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20123.log.gz 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29077.log 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29627.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28355.log.gz 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29207.log 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29627.log.gz 2024-04-25T21:41:20.511 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29077.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28366.log 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29077.log.gz 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29207.log.gz 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28867.log 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29497.log 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr: 91.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28366.log: /var/log/ceph/ceph-client.admin.28867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29885.log 2024-04-25T21:41:20.512 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28366.log.gz/var/log/ceph/ceph-client.admin.29497.log: 2024-04-25T21:41:20.513 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28365.log 2024-04-25T21:41:20.513 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28867.log.gz 2024-04-25T21:41:20.513 INFO:teuthology.orchestra.run.smithi006.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29497.log.gz 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29885.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29885.log.gz 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28510.log 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28700.log 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28365.log.gz 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28510.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29679.log 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28700.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28510.log.gz 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28295.log 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28700.log.gz 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28945.log 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28364.log 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29679.log.gz 2024-04-25T21:41:20.516 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28295.log: /var/log/ceph/ceph-client.admin.28945.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28295.log.gz 2024-04-25T21:41:20.517 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28945.log.gz 2024-04-25T21:41:20.517 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29575.log 2024-04-25T21:41:20.517 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28364.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T21:41:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29575.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28364.log.gz 2024-04-25T21:41:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29367.log 2024-04-25T21:41:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29263.log 2024-04-25T21:41:20.518 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.29367.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29575.log.gz 2024-04-25T21:41:20.518 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29367.log.gz 2024-04-25T21:41:20.518 INFO:teuthology.orchestra.run.smithi006.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T21:41:20.519 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29289.log 2024-04-25T21:41:20.520 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23744.log 2024-04-25T21:41:20.520 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29289.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.29263.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29471.log 2024-04-25T21:41:20.520 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:20.520 INFO:teuthology.orchestra.run.smithi006.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29289.log.gz 2024-04-25T21:41:20.520 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.23744.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.23744.log.gz 2024-04-25T21:41:20.521 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28659.log 2024-04-25T21:41:20.521 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29393.log 2024-04-25T21:41:20.522 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28659.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.29471.log.gz --verbose 2024-04-25T21:41:20.522 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.29653.log 2024-04-25T21:41:20.522 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28659.log.gz 2024-04-25T21:41:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29393.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29393.log.gz 2024-04-25T21:41:20.523 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29549.log 2024-04-25T21:41:20.524 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24031.log 2024-04-25T21:41:20.525 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28893.log 2024-04-25T21:41:20.525 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29653.log: /var/log/ceph/ceph-client.admin.29549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28269.log 2024-04-25T21:41:20.525 INFO:teuthology.orchestra.run.smithi006.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29653.log.gz 2024-04-25T21:41:20.525 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28893.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29549.log.gz 2024-04-25T21:41:20.525 INFO:teuthology.orchestra.run.smithi006.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28893.log.gz 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28815.log 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29757.log 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28815.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29783.log 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28269.log.gz 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29757.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28815.log.gz 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29315.log 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29757.log.gz 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29445.log 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29731.log 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29783.log.gz 2024-04-25T21:41:20.534 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29315.log: /var/log/ceph/ceph-client.admin.29445.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29315.log.gz 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29445.log.gz 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23864.log 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29601.log 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.23864.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28243.log 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29731.log.gz 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.23864.log.gz 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29601.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29601.log.gz 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29155.log 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.28243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29809.log 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29155.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28243.log.gz 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.29809.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29155.log.gz 2024-04-25T21:41:20.535 INFO:teuthology.orchestra.run.smithi006.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29809.log.gz 2024-04-25T21:41:20.536 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23952.log 2024-04-25T21:41:20.536 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23928.log 2024-04-25T21:41:20.536 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23952.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.24031.log.gz 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23983.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.23952.log.gz 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23928.log: /var/log/ceph/ceph-client.admin.23983.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.3.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.23983.log.gz 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28171.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.23928.log.gz 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.28171.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24007.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28171.log.gz 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23904.log 2024-04-25T21:41:20.537 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28204.log 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr: 84.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23904.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.24007.log.gz 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.28204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28170.log 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.23904.log.gz 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.28170.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr: 88.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28206.log 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28205.log 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.28206.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.28204.log.gz 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28206.log.gz 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr: 90.7%/var/log/ceph/ceph-client.admin.28205.log: -- replaced with /var/log/ceph/ceph-client.admin.28170.log.gz 2024-04-25T21:41:20.538 INFO:teuthology.orchestra.run.smithi203.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28205.log.gz 2024-04-25T21:41:20.567 INFO:teuthology.orchestra.run.smithi203.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T21:41:20.574 INFO:teuthology.orchestra.run.smithi006.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T21:41:20.588 INFO:teuthology.orchestra.run.smithi203.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T21:41:20.604 INFO:teuthology.orchestra.run.smithi006.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T21:41:20.674 INFO:teuthology.orchestra.run.smithi203.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T21:41:21.042 INFO:teuthology.orchestra.run.smithi203.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T21:41:21.049 INFO:teuthology.orchestra.run.smithi203.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T21:41:21.050 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T21:41:21.050 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m0.544s 2024-04-25T21:41:21.050 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m1.248s 2024-04-25T21:41:21.050 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.082s 2024-04-25T21:41:21.054 INFO:teuthology.orchestra.run.smithi006.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T21:41:21.077 INFO:teuthology.orchestra.run.smithi006.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T21:41:21.080 INFO:teuthology.orchestra.run.smithi006.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T21:41:21.081 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-04-25T21:41:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m0.620s 2024-04-25T21:41:21.081 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m1.906s 2024-04-25T21:41:21.082 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.110s 2024-04-25T21:41:21.082 INFO:tasks.ceph:Archiving logs... 2024-04-25T21:41:21.083 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/remote/smithi006/log 2024-04-25T21:41:21.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T21:41:21.395 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/remote/smithi203/log 2024-04-25T21:41:21.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T21:41:21.563 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T21:41:21.571 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T21:41:21.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T21:41:21.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T21:41:21.646 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T21:41:21.654 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T21:41:21.655 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T21:41:21.657 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T21:41:21.668 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T21:41:21.668 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-04-25T21:41:21.668 INFO:teuthology.orchestra.run.smithi006.stdout:*hv01.front.sepi 67.205.162.81 3 u 43 64 17 0.146 -2.164 0.267 2024-04-25T21:41:21.668 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi 63.231.80.2 3 u 42 64 17 0.144 -0.393 0.841 2024-04-25T21:41:21.668 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi 74.6.168.72 3 u 41 64 17 0.076 -1.930 0.138 2024-04-25T21:41:21.668 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:41:21.669 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T21:41:21.669 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-04-25T21:41:21.670 INFO:teuthology.orchestra.run.smithi203.stdout: hv01.front.sepi 67.205.162.81 3 u 46 64 17 0.228 -3.034 1.842 2024-04-25T21:41:21.670 INFO:teuthology.orchestra.run.smithi203.stdout: hv02.front.sepi 63.231.80.2 3 u 43 64 17 0.132 -1.031 1.854 2024-04-25T21:41:21.670 INFO:teuthology.orchestra.run.smithi203.stdout:*hv03.front.sepi 74.6.168.72 3 u 48 64 17 0.135 -2.896 1.790 2024-04-25T21:41:21.670 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T21:41:21.671 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T21:41:21.773 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T21:41:21.773 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T21:41:21.804 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T21:41:21.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T21:41:21.822 INFO:teuthology.task.internal:Duration was 679.954799 seconds 2024-04-25T21:41:21.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T21:41:21.830 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T21:41:21.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T21:41:21.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T21:41:21.870 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T21:41:21.870 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-04-25T21:41:21.870 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T21:41:21.920 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T21:41:21.920 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T21:41:21.930 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T21:41:21.930 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T21:41:21.964 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T21:41:21.994 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T21:41:21.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T21:41:22.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T21:41:22.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T21:41:22.107 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T21:41:22.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T21:41:22.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T21:41:22.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T21:41:22.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T21:41:22.168 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T21:41:22.177 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-04-25T21:41:22.184 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-25T21:41:22.299 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T21:41:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:41:22.304 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T21:41:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T21:41:22.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T21:41:22.357 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T21:41:22.357 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/remote/smithi006 2024-04-25T21:41:22.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T21:41:22.378 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673740/remote/smithi203 2024-04-25T21:41:22.378 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T21:41:22.410 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T21:41:22.410 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T21:41:22.416 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T21:41:22.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T21:41:22.466 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T21:41:22.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T21:41:22.475 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T21:41:22.475 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T21:41:22.477 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T21:41:22.481 INFO:teuthology.orchestra.run.smithi006.stdout: 658637 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 21:41 /home/ubuntu/cephtest 2024-04-25T21:41:22.503 INFO:teuthology.orchestra.run.smithi203.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 21:41 /home/ubuntu/cephtest 2024-04-25T21:41:22.505 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T21:41:22.515 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T21:41:22.536 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925} duration: 679.9547986984253 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T21:41:22.536 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T21:41:22.608 INFO:teuthology.run:pass